Skip to contentSkip to conversation

kdevops: add milvus with minio support

PATCH In review Luis Chamberlain 3 participants 8 messages Aug 27, 2025 – Sep 1, 2025 Kbuild

lore
Share & tools
lore
#0 lore raw

cover kdevops: add milvus with minio support

This adds the ability to test milvus on minio with different filesystem configuration targets. There's a basic configuration you can run which will just support one default filesystem, which will be used where you place your docker image and also where we place the minio instance. Then there is multifs support where just as with fstests support on kdevops you can select a slew of different filesystem targets to try to test.

Recommendation is to stick to 40 iterations at 1,000,000 tests unless you have more than 100 GiB per guest to spare. If you have space to spare then you know how to ballpark it.

On High Capacity SSDs, the world is our oyster.

You can see a demo of results here:

https://github.com/mcgrof/demo-milvus-kdevops-results

These are just demos. On guests. Nothing really useful. I should point out this has AB testing automated as well so we can leverage this to test for instance ... parallel writeback in an automated way ;)

If you want to test this you can also use this branch on kdevops:

https://github.com/linux-kdevops/kdevops/tree/mcgrof/20250827-milvus

I am in hopes someone will just prompt an AI for bare metal support while I sleep. It should be... easy. Just create the partitions already, use one host and ask the prompt to not mkfs for you. So don't use multi-fs support at first. Just use the option to create the storage partition where you place docker. In fact you can copy and paste this prompt the the AI, and I think it will know what to do. You just skip some steps as the filesystems can be created and mounted for you. You just need the host file created by you manually for the target node. That and infer user and group id support (WORKFLOW_INFER_USER_AND_GROUP).

Luis Chamberlain (2):
ai: add Milvus vector database benchmarking support
ai: add multi-filesystem testing support for Milvus benchmarks

 .github/workflows/docker-tests.yml            |    6 +
 .gitignore                                    |    3 +-
 Makefile                                      |    2 +-
 README.md                                     |   18 +
 defconfigs/ai-milvus-docker                   |  113 ++
 defconfigs/ai-milvus-docker-ci                |   51 +
 defconfigs/ai-milvus-multifs                  |   67 +
 defconfigs/ai-milvus-multifs-distro           |  109 ++
 defconfigs/ai-milvus-multifs-extended         |  108 ++
 docs/ai/README.md                             |  108 ++
 docs/ai/vector-databases/README.md            |   75 +
 docs/ai/vector-databases/milvus.md            |  264 +++
 kconfigs/workflows/Kconfig                    |   27 +
 playbooks/ai.yml                              |   11 +
 playbooks/ai_benchmark.yml                    |    8 +
 playbooks/ai_destroy.yml                      |   24 +
 playbooks/ai_install.yml                      |   14 +
 playbooks/ai_multifs.yml                      |   24 +
 playbooks/ai_results.yml                      |    6 +
 playbooks/ai_setup.yml                        |    6 +
 playbooks/ai_tests.yml                        |   31 +
 playbooks/ai_uninstall.yml                    |    6 +
 .../debian13-ai-btrfs-default-dev.yml         |    8 +
 .../host_vars/debian13-ai-btrfs-default.yml   |    8 +
 .../debian13-ai-ext4-16k-bigalloc-dev.yml     |    8 +
 .../debian13-ai-ext4-16k-bigalloc.yml         |    8 +
 .../host_vars/debian13-ai-ext4-4k-dev.yml     |    8 +
 playbooks/host_vars/debian13-ai-ext4-4k.yml   |    8 +
 .../host_vars/debian13-ai-xfs-16k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-16k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-4k-4ks-dev.yml  |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks.yml     |   10 +
 .../files/analyze_results.py                  | 1701 +++++++++++++++++
 .../files/generate_better_graphs.py           |  550 ++++++
 .../files/generate_graphs.py                  |  362 ++++
 .../files/generate_html_report.py             |  610 ++++++
 .../roles/ai_collect_results/tasks/main.yml   |  202 ++
 .../templates/analysis_config.json.j2         |    6 +
 playbooks/roles/ai_destroy/tasks/main.yml     |   63 +
 .../roles/ai_docker_storage/tasks/main.yml    |  123 ++
 playbooks/roles/ai_install/tasks/main.yml     |   90 +
 .../roles/ai_milvus_storage/tasks/main.yml    |  161 ++
 .../tasks/generate_comparison.yml             |  279 +++
 playbooks/roles/ai_multifs_run/tasks/main.yml |   23 +
 .../tasks/run_single_filesystem.yml           |  104 +
 .../templates/milvus_config.json.j2           |   42 +
 .../roles/ai_multifs_setup/defaults/main.yml  |   49 +
 .../roles/ai_multifs_setup/tasks/main.yml     |   70 +
 playbooks/roles/ai_results/tasks/main.yml     |   22 +
 .../files/milvus_benchmark.py                 |  556 ++++++
 .../roles/ai_run_benchmarks/tasks/main.yml    |  181 ++
 .../templates/benchmark_config.json.j2        |   24 +
 playbooks/roles/ai_setup/tasks/main.yml       |  115 ++
 playbooks/roles/ai_uninstall/tasks/main.yml   |   62 +
 playbooks/roles/gen_hosts/tasks/main.yml      |   33 +
 .../roles/gen_hosts/templates/fstests.j2      |    2 +
 playbooks/roles/gen_hosts/templates/gitr.j2   |    2 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |   99 +
 .../roles/gen_hosts/templates/nfstest.j2      |    2 +
 playbooks/roles/gen_hosts/templates/pynfs.j2  |    2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  124 ++
 .../roles/guestfs/tasks/bringup/main.yml      |   15 +
 playbooks/roles/milvus/README.md              |  181 ++
 playbooks/roles/milvus/defaults/main.yml      |   74 +
 .../roles/milvus/files/milvus_benchmark.py    |  348 ++++
 playbooks/roles/milvus/files/milvus_utils.py  |  134 ++
 playbooks/roles/milvus/meta/main.yml          |   30 +
 playbooks/roles/milvus/tasks/benchmark.yml    |   61 +
 .../roles/milvus/tasks/benchmark_setup.yml    |   58 +
 .../roles/milvus/tasks/install_docker.yml     |   97 +
 playbooks/roles/milvus/tasks/main.yml         |   52 +
 playbooks/roles/milvus/tasks/setup.yml        |  107 ++
 .../milvus/templates/benchmark_config.json.j2 |   25 +
 .../templates/docker-compose.override.yml.j2  |   24 +
 .../milvus/templates/docker-compose.yml.j2    |   64 +
 .../roles/milvus/templates/milvus.yaml.j2     |   30 +
 .../milvus/templates/test_connection.py.j2    |   25 +
 scripts/guestfs.Makefile                      |    2 +-
 workflows/Makefile                            |    4 +
 workflows/ai/Kconfig                          |  177 ++
 workflows/ai/Kconfig.docker                   |  172 ++
 workflows/ai/Kconfig.docker-storage           |  201 ++
 workflows/ai/Kconfig.fs                       |  118 ++
 workflows/ai/Kconfig.multifs                  |  184 ++
 workflows/ai/Kconfig.native                   |  184 ++
 workflows/ai/Makefile                         |  160 ++
 workflows/ai/scripts/analysis_config.json     |    6 +
 workflows/ai/scripts/analyze_results.py       | 1701 +++++++++++++++++
 workflows/ai/scripts/generate_graphs.py       |  362 ++++
 workflows/ai/scripts/generate_html_report.py  |  610 ++++++
 93 files changed, 12061 insertions(+), 3 deletions(-)
 create mode 100644 defconfigs/ai-milvus-docker
 create mode 100644 defconfigs/ai-milvus-docker-ci
 create mode 100644 defconfigs/ai-milvus-multifs
 create mode 100644 defconfigs/ai-milvus-multifs-distro
 create mode 100644 defconfigs/ai-milvus-multifs-extended
 create mode 100644 docs/ai/README.md
 create mode 100644 docs/ai/vector-databases/README.md
 create mode 100644 docs/ai/vector-databases/milvus.md
 create mode 100644 playbooks/ai.yml
 create mode 100644 playbooks/ai_benchmark.yml
 create mode 100644 playbooks/ai_destroy.yml
 create mode 100644 playbooks/ai_install.yml
 create mode 100644 playbooks/ai_multifs.yml
 create mode 100644 playbooks/ai_results.yml
 create mode 100644 playbooks/ai_setup.yml
 create mode 100644 playbooks/ai_tests.yml
 create mode 100644 playbooks/ai_uninstall.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks.yml
 create mode 100755 playbooks/roles/ai_collect_results/files/analyze_results.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_better_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_html_report.py
 create mode 100644 playbooks/roles/ai_collect_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_collect_results/templates/analysis_config.json.j2
 create mode 100644 playbooks/roles/ai_destroy/tasks/main.yml
 create mode 100644 playbooks/roles/ai_docker_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_install/tasks/main.yml
 create mode 100644 playbooks/roles/ai_milvus_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml
 create mode 100644 playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2
 create mode 100644 playbooks/roles/ai_multifs_setup/defaults/main.yml
 create mode 100644 playbooks/roles/ai_multifs_setup/tasks/main.yml
 create mode 100644 playbooks/roles/ai_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/ai_run_benchmarks/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/ai_setup/tasks/main.yml
 create mode 100644 playbooks/roles/ai_uninstall/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/README.md
 create mode 100644 playbooks/roles/milvus/defaults/main.yml
 create mode 100644 playbooks/roles/milvus/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/milvus/files/milvus_utils.py
 create mode 100644 playbooks/roles/milvus/meta/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark_setup.yml
 create mode 100644 playbooks/roles/milvus/tasks/install_docker.yml
 create mode 100644 playbooks/roles/milvus/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/setup.yml
 create mode 100644 playbooks/roles/milvus/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.override.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/milvus.yaml.j2
 create mode 100644 playbooks/roles/milvus/templates/test_connection.py.j2
 create mode 100644 workflows/ai/Kconfig
 create mode 100644 workflows/ai/Kconfig.docker
 create mode 100644 workflows/ai/Kconfig.docker-storage
 create mode 100644 workflows/ai/Kconfig.fs
 create mode 100644 workflows/ai/Kconfig.multifs
 create mode 100644 workflows/ai/Kconfig.native
 create mode 100644 workflows/ai/Makefile
 create mode 100644 workflows/ai/scripts/analysis_config.json
 create mode 100755 workflows/ai/scripts/analyze_results.py
 create mode 100755 workflows/ai/scripts/generate_graphs.py
 create mode 100755 workflows/ai/scripts/generate_html_report.py
Signature · 1 line
2.50.1
#1 lore raw

PATCH 1/2 ai: add Milvus vector database benchmarking support

Add initial AI/ML workflow infrastructure starting with Milvus vector database benchmarking. This provides a foundation for testing AI systems with the same rigor as existing kernel testing workflows (fstests, blktests).

Key features:
- Docker-based Milvus deployment with etcd and MinIO
- Supports using a dedicated drive for docker /var/lib/docker/ including custom filesystem configurations
- Python virtual environment management for benchmark dependencies
- Comprehensive benchmarking of vector operations (insert, search, delete)
- A/B testing support for baseline vs development comparisons
- Performance visualization focusing on key metrics (QPS, latency)
- Result collection and analysis infrastructure

Performance Metrics:
The benchmarks focus on two critical vector database metrics:
- QPS (Queries Per Second): Throughput measurement for search operations
- Latency: Response time percentiles (p50, p95, p99) for operations

Recall rate measurement is challenging without ground truth data - the correct answers must be known beforehand to measure search accuracy. Since we generate random vectors for testing, establishing meaningful ground truth would require careful similarity calculations that would essentially duplicate the work being tested.

Defconfigs:
- ai-milvus-docker: Standard Docker-based Milvus deployment
- ai-milvus-docker-ci: CI-optimized with minimal dataset (1000 vectors)

Workflow integration follows kdevops patterns:
make defconfig-ai-milvus-docker
make bringup
make ai # Setup infrastructure
make ai-tests # Run benchmarks
make ai-results # View results

The implementation handles proper cleanup, lock file management, and comprehensive error handling to ensure reliable benchmark execution.

Generated-by Claude AI
---
 .gitignore                                    |    3 +-
 README.md                                     |   18 +
 defconfigs/ai-milvus-docker                   |  113 ++
 defconfigs/ai-milvus-docker-ci                |   51 +
 docs/ai/README.md                             |  108 ++
 docs/ai/vector-databases/README.md            |   76 ++
 docs/ai/vector-databases/milvus.md            |  264 ++++
 kconfigs/workflows/Kconfig                    |   27 +
 playbooks/ai.yml                              |   11 +
 playbooks/ai_benchmark.yml                    |    8 +
 playbooks/ai_destroy.yml                      |   24 +
 playbooks/ai_install.yml                      |    8 +
 playbooks/ai_results.yml                      |    6 +
 playbooks/ai_setup.yml                        |    6 +
 playbooks/ai_tests.yml                        |   31 +
 playbooks/ai_uninstall.yml                    |    6 +
 .../debian13-ai-btrfs-default-dev.yml         |    8 +
 .../host_vars/debian13-ai-btrfs-default.yml   |    8 +
 .../debian13-ai-ext4-16k-bigalloc-dev.yml     |    8 +
 .../debian13-ai-ext4-16k-bigalloc.yml         |    8 +
 .../host_vars/debian13-ai-ext4-4k-dev.yml     |    8 +
 playbooks/host_vars/debian13-ai-ext4-4k.yml   |    8 +
 .../host_vars/debian13-ai-xfs-16k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-16k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-4k-4ks-dev.yml  |   10 +
 .../host_vars/debian13-ai-xfs-4k-4ks.yml      |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks.yml     |   10 +
 .../files/analyze_results.py                  |  979 ++++++++++++++
 .../files/generate_better_graphs.py           |  548 ++++++++
 .../files/generate_graphs.py                  |  678 ++++++++++
 .../files/generate_html_report.py             |  427 ++++++
 .../roles/ai_collect_results/tasks/main.yml   |  220 +++
 .../templates/analysis_config.json.j2         |    6 +
 playbooks/roles/ai_destroy/tasks/main.yml     |   63 +
 .../roles/ai_docker_storage/tasks/main.yml    |  123 ++
 playbooks/roles/ai_install/tasks/main.yml     |   90 ++
 playbooks/roles/ai_results/tasks/main.yml     |   22 +
 .../files/milvus_benchmark.py                 |  506 +++++++
 .../roles/ai_run_benchmarks/tasks/main.yml    |  181 +++
 .../templates/benchmark_config.json.j2        |   24 +
 playbooks/roles/ai_setup/tasks/main.yml       |  115 ++
 playbooks/roles/ai_uninstall/tasks/main.yml   |   62 +
 playbooks/roles/gen_hosts/tasks/main.yml      |   14 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |  108 ++
 playbooks/roles/gen_nodes/tasks/main.yml      |   34 +
 playbooks/roles/milvus/README.md              |  181 +++
 playbooks/roles/milvus/defaults/main.yml      |   74 ++
 .../roles/milvus/files/milvus_benchmark.py    |  348 +++++
 playbooks/roles/milvus/files/milvus_utils.py  |  134 ++
 playbooks/roles/milvus/meta/main.yml          |   30 +
 playbooks/roles/milvus/tasks/benchmark.yml    |   61 +
 .../roles/milvus/tasks/benchmark_setup.yml    |   58 +
 .../roles/milvus/tasks/install_docker.yml     |   97 ++
 playbooks/roles/milvus/tasks/main.yml         |   52 +
 playbooks/roles/milvus/tasks/setup.yml        |  107 ++
 .../milvus/templates/benchmark_config.json.j2 |   25 +
 .../templates/docker-compose.override.yml.j2  |   24 +
 .../milvus/templates/docker-compose.yml.j2    |   64 +
 .../roles/milvus/templates/milvus.yaml.j2     |   30 +
 .../milvus/templates/test_connection.py.j2    |   25 +
 workflows/Makefile                            |    4 +
 workflows/ai/Kconfig                          |  164 +++
 workflows/ai/Kconfig.docker                   |  172 +++
 workflows/ai/Kconfig.docker-storage           |  201 +++
 workflows/ai/Kconfig.native                   |  184 +++
 workflows/ai/Makefile                         |  160 +++
 workflows/ai/scripts/analysis_config.json     |    6 +
 workflows/ai/scripts/analyze_results.py       |  979 ++++++++++++++
 workflows/ai/scripts/generate_graphs.py       | 1174 +++++++++++++++++
 workflows/ai/scripts/generate_html_report.py  |  558 ++++++++
 73 files changed, 9999 insertions(+), 1 deletion(-)
 create mode 100644 defconfigs/ai-milvus-docker
 create mode 100644 defconfigs/ai-milvus-docker-ci
 create mode 100644 docs/ai/README.md
 create mode 100644 docs/ai/vector-databases/README.md
 create mode 100644 docs/ai/vector-databases/milvus.md
 create mode 100644 playbooks/ai.yml
 create mode 100644 playbooks/ai_benchmark.yml
 create mode 100644 playbooks/ai_destroy.yml
 create mode 100644 playbooks/ai_install.yml
 create mode 100644 playbooks/ai_results.yml
 create mode 100644 playbooks/ai_setup.yml
 create mode 100644 playbooks/ai_tests.yml
 create mode 100644 playbooks/ai_uninstall.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks.yml
 create mode 100755 playbooks/roles/ai_collect_results/files/analyze_results.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_better_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_html_report.py
 create mode 100644 playbooks/roles/ai_collect_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_collect_results/templates/analysis_config.json.j2
 create mode 100644 playbooks/roles/ai_destroy/tasks/main.yml
 create mode 100644 playbooks/roles/ai_docker_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_install/tasks/main.yml
 create mode 100644 playbooks/roles/ai_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/ai_run_benchmarks/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/ai_setup/tasks/main.yml
 create mode 100644 playbooks/roles/ai_uninstall/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/README.md
 create mode 100644 playbooks/roles/milvus/defaults/main.yml
 create mode 100644 playbooks/roles/milvus/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/milvus/files/milvus_utils.py
 create mode 100644 playbooks/roles/milvus/meta/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark_setup.yml
 create mode 100644 playbooks/roles/milvus/tasks/install_docker.yml
 create mode 100644 playbooks/roles/milvus/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/setup.yml
 create mode 100644 playbooks/roles/milvus/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.override.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/milvus.yaml.j2
 create mode 100644 playbooks/roles/milvus/templates/test_connection.py.j2
 create mode 100644 workflows/ai/Kconfig
 create mode 100644 workflows/ai/Kconfig.docker
 create mode 100644 workflows/ai/Kconfig.docker-storage
 create mode 100644 workflows/ai/Kconfig.native
 create mode 100644 workflows/ai/Makefile
 create mode 100644 workflows/ai/scripts/analysis_config.json
 create mode 100755 workflows/ai/scripts/analyze_results.py
 create mode 100755 workflows/ai/scripts/generate_graphs.py
 create mode 100755 workflows/ai/scripts/generate_html_report.py
.gitignore +2 -1
index e5a13676..75e4712d 100644
@@ -32,7 +32,6 @@ scripts/workflows/fstests/lib/__pycache__/
 scripts/workflows/blktests/lib/__pycache__/
 scripts/workflows/lib/__pycache__/
 
-
 include/
 
 # You can override role specific stuff on these
@@ -48,7 +47,9 @@ playbooks/secret.yml
 playbooks/python/workflows/fstests/__pycache__/
 playbooks/python/workflows/fstests/lib/__pycache__/
 playbooks/python/workflows/fstests/gen_results_summary.pyc
+playbooks/roles/ai_run_benchmarks/files/__pycache__/
 
+workflows/ai/results/
 workflows/pynfs/results/
 
 workflows/fstests/new_expunge_files.txt
README.md +18 -0
index 0c30762a..cb5fbc1f 100644
@@ -14,6 +14,7 @@ Table of Contents
       * [reboot-limit](#reboot-limit)
       * [sysbench](#sysbench)
       * [fio-tests](#fio-tests)
+      * [AI workflow](#ai-workflow)
    * [kdevops chats](#kdevops-chats)
    * [kdevops on discord](#kdevops-on-discord)
       * [kdevops IRC](#kdevops-irc)
@@ -273,6 +274,22 @@ A/B testing capabilities, and advanced graphing and visualization support. For
 detailed configuration and usage information, refer to the
 [kdevops fio-tests documentation](docs/fio-tests.md).
 
+### AI workflow
+
+kdevops now supports AI/ML system benchmarking, starting with vector databases
+like Milvus. Similar to fstests, you can quickly set up and benchmark AI
+infrastructure with just a few commands:
+
+```bash
+make defconfig-ai-milvus-docker
+make bringup
+make ai
+```
+
+The AI workflow supports A/B testing, filesystem performance impact analysis,
+and comprehensive benchmarking of vector similarity search workloads. For
+details, see the [kdevops AI workflow documentation](docs/ai/README.md).
+
 ## kdevops chats
 
 We use discord and IRC. Right now we have more folks on discord than on IRC.
@@ -324,6 +341,7 @@ want to just use the kernel that comes with your Linux distribution.
   * [kdevops NFS docs](docs/nfs.md)
   * [kdevops selftests docs](docs/selftests.md)
   * [kdevops reboot-limit docs](docs/reboot-limit.md)
+  * [kdevops AI workflow docs](docs/ai/README.md)
 
 # kdevops general documentation
 
defconfigs/ai-milvus-docker new file +113 -0

113 lines not on this page · raw message

defconfigs/ai-milvus-docker-ci new file +51 -0

51 lines not on this page · raw message

docs/ai/README.md new file +108 -0

108 lines not on this page · raw message

docs/ai/vector-databases/README.md new file +76 -0

76 lines not on this page · raw message

docs/ai/vector-databases/milvus.md new file +264 -0

264 lines not on this page · raw message

kconfigs/workflows/Kconfig +27 -0
index 6b2a3769..70898a1a 100644
@@ -214,6 +214,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS
 	  This will dedicate your configuration to running only the
 	  fio-tests workflow for comprehensive storage performance testing.
 
+config KDEVOPS_WORKFLOW_DEDICATE_AI
+	bool "ai"
+	select KDEVOPS_WORKFLOW_ENABLE_AI
+	help
+	  This will dedicate your configuration to running only the
+	  AI workflow for vector database performance testing.
+
 endchoice
 
 config KDEVOPS_WORKFLOW_NAME
@@ -229,6 +236,7 @@ config KDEVOPS_WORKFLOW_NAME
 	default "sysbench" if KDEVOPS_WORKFLOW_DEDICATE_SYSBENCH
 	default "mmtests" if KDEVOPS_WORKFLOW_DEDICATE_MMTESTS
 	default "fio-tests" if KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS
+	default "ai" if KDEVOPS_WORKFLOW_DEDICATE_AI
 
 endif
 
@@ -338,6 +346,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FIO_TESTS
 	  Select this option if you want to provision fio-tests on a
 	  single target node for by-hand testing.
 
+config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_AI
+	bool "ai"
+	select KDEVOPS_WORKFLOW_ENABLE_AI
+	depends on LIBVIRT || TERRAFORM_PRIVATE_NET
+	help
+	  Select this option if you want to provision AI benchmarks on a
+	  single target node for by-hand testing.
+
 endif # !WORKFLOWS_DEDICATED_WORKFLOW
 
 config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
@@ -462,6 +478,17 @@ source "workflows/fio-tests/Kconfig"
 endmenu
 endif # KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
 
+config KDEVOPS_WORKFLOW_ENABLE_AI
+	bool
+	output yaml
+	default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_AI || KDEVOPS_WORKFLOW_DEDICATE_AI
+
+if KDEVOPS_WORKFLOW_ENABLE_AI
+menu "Configure and run AI benchmarks"
+source "workflows/ai/Kconfig"
+endmenu
+endif # KDEVOPS_WORKFLOW_ENABLE_AI
+
 config KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE
        bool "Attain SSD steady state prior to tests"
        output yaml
playbooks/ai.yml new file +11 -0
new file mode 100644
index 00000000..b1613309
@@ -0,0 +1,11 @@
+---
+# Main AI workflow orchestration playbook
+# This demonstrates the scalable structure for AI workflows
+
+- name: AI Workflow - Vector Database Setup
+  ansible.builtin.import_playbook: ai_install.yml
+  when: ai_workflow_vector_db | default(true) | bool
+  tags: ['ai', 'setup']
+
+# Benchmarks are run separately via make ai-tests targets
+# They should not run during the setup phase (make ai)
playbooks/ai_benchmark.yml new file +8 -0
new file mode 100644
index 00000000..85fc117c
@@ -0,0 +1,8 @@
+---
+- name: Run Milvus Vector Database Benchmarks
+  hosts: ai
+  vars:
+    ai_vector_db_milvus_benchmark_enable: true
+  roles:
+    - role: milvus
+      tags: ['ai', 'vector_db', 'milvus', 'benchmark']
playbooks/ai_destroy.yml new file +24 -0
new file mode 100644
index 00000000..eef07b2a
@@ -0,0 +1,24 @@
+---
+- name: Destroy Milvus Vector Database
+  hosts: ai
+  become: true
+  tasks:
+    - name: Stop Milvus containers
+      community.docker.docker_compose:
+        project_src: "{{ ai_vector_db_milvus_config_dir }}"
+        state: absent
+      when: ai_vector_db_milvus_docker | bool
+      # TODO: Review - was ignore_errors: true
+      failed_when: false  # Always succeed - review this condition
+
+    - name: Remove Milvus data directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - "{{ ai_vector_db_milvus_data_dir }}"
+        - "{{ ai_vector_db_milvus_config_dir }}"
+        - "{{ ai_vector_db_milvus_log_dir }}"
+      when: ai_vector_db_force_destroy | default(false) | bool
+
+  tags: ['ai', 'vector_db', 'milvus', 'destroy']
playbooks/ai_install.yml new file +8 -0
new file mode 100644
index 00000000..70b734e4
@@ -0,0 +1,8 @@
+---
+- name: Install Milvus Vector Database
+  hosts: ai
+  become: true
+  become_user: root
+  roles:
+    - role: milvus
+      tags: ['ai', 'vector_db', 'milvus', 'install']
playbooks/ai_results.yml new file +6 -0
new file mode 100644
index 00000000..881295eb
@@ -0,0 +1,6 @@
+---
+- name: Collect and analyze AI benchmark results
+  hosts: ai
+  roles:
+    - ai_collect_results
+  tags: ['ai', 'ai_results']
playbooks/ai_setup.yml new file +6 -0
new file mode 100644
index 00000000..f0007ee2
@@ -0,0 +1,6 @@
+---
+- name: Setup AI benchmark environment
+  hosts: ai
+  roles:
+    - ai_setup
+  tags: ['ai', 'ai_setup']
playbooks/ai_tests.yml new file +31 -0
new file mode 100644
index 00000000..1a5638fc
@@ -0,0 +1,31 @@
+---
+# AI Tests/Benchmarks playbook
+# This ensures AI infrastructure is setup before running benchmarks
+
+- name: AI Tests - Ensure Milvus is installed
+  hosts: ai
+  become: true
+  become_user: root
+  roles:
+    - role: milvus
+      when: ai_vector_db_milvus | default(false) | bool
+      tags: ['ai', 'milvus', 'setup']
+
+- name: AI Tests - Vector Database Benchmarks
+  hosts: ai
+  become: true
+  vars:
+    # Skip infrastructure setup when running tests
+    ai_skip_setup: true
+  roles:
+    - role: ai_run_benchmarks
+      when: ai_vector_db_milvus | default(false) | bool
+      tags: ['ai', 'benchmark']
+
+- name: AI Tests - Results Collection
+  hosts: ai
+  become: true
+  roles:
+    - role: ai_collect_results
+      when: ai_collect_results | default(true) | bool
+      tags: ['ai', 'results']
playbooks/ai_uninstall.yml new file +6 -0
new file mode 100644
index 00000000..fb537664
@@ -0,0 +1,6 @@
+---
+- name: Uninstall AI benchmark components
+  hosts: ai
+  roles:
+    - ai_uninstall
+  tags: ['ai', 'ai_uninstall']
playbooks/host_vars/debian13-ai-btrfs-default-dev.yml new file +8 -0
new file mode 100644
index 00000000..85b95a52
@@ -0,0 +1,8 @@
+---
+# btrfs default configuration (dev)
+ai_docker_fstype: "btrfs"
+ai_docker_btrfs_mkfs_opts: "-f"
+filesystem_type: "btrfs"
+filesystem_block_size: "default"
+ai_filesystem: "btrfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-btrfs-default.yml new file +8 -0
new file mode 100644
index 00000000..f4f18b9e
@@ -0,0 +1,8 @@
+---
+# btrfs default configuration
+ai_docker_fstype: "btrfs"
+ai_docker_btrfs_mkfs_opts: "-f"
+filesystem_type: "btrfs"
+filesystem_block_size: "default"
+ai_filesystem: "btrfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-ext4-16k-bigalloc-dev.yml new file +8 -0
new file mode 100644
index 00000000..e4b1a9da
@@ -0,0 +1,8 @@
+---
+# ext4 16k bigalloc configuration (dev)
+ai_docker_fstype: "ext4"
+ai_docker_ext4_mkfs_opts: "-b 4096 -C 16384 -O bigalloc"
+filesystem_type: "ext4"
+filesystem_block_size: "16k-bigalloc"
+ai_filesystem: "ext4"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-ext4-16k-bigalloc.yml new file +8 -0
new file mode 100644
index 00000000..a5624440
@@ -0,0 +1,8 @@
+---
+# ext4 16k bigalloc configuration
+ai_docker_fstype: "ext4"
+ai_docker_ext4_mkfs_opts: "-b 4096 -C 16384 -O bigalloc"
+filesystem_type: "ext4"
+filesystem_block_size: "16k-bigalloc"
+ai_filesystem: "ext4"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-ext4-4k-dev.yml new file +8 -0
new file mode 100644
index 00000000..6ca5fec5
@@ -0,0 +1,8 @@
+---
+# ext4 4k block configuration (dev)
+ai_docker_fstype: "ext4"
+ai_docker_ext4_mkfs_opts: "-b 4096"
+filesystem_type: "ext4"
+filesystem_block_size: "4k"
+ai_filesystem: "ext4"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-ext4-4k.yml new file +8 -0
new file mode 100644
index 00000000..f2840faa
@@ -0,0 +1,8 @@
+---
+# ext4 4k block configuration
+ai_docker_fstype: "ext4"
+ai_docker_ext4_mkfs_opts: "-b 4096"
+filesystem_type: "ext4"
+filesystem_block_size: "4k"
+ai_filesystem: "ext4"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-16k-4ks-dev.yml new file +10 -0
new file mode 100644
index 00000000..429e6461
@@ -0,0 +1,10 @@
+---
+# XFS 16k block, 4k sector configuration (dev)
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 16384
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "16k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-16k-4ks.yml new file +10 -0
new file mode 100644
index 00000000..15200810
@@ -0,0 +1,10 @@
+---
+# XFS 16k block, 4k sector configuration  
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 16384
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "16k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-32k-4ks-dev.yml new file +10 -0
new file mode 100644
index 00000000..6f30a053
@@ -0,0 +1,10 @@
+---
+# XFS 32k block, 4k sector configuration (dev)
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 32768
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "32k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-32k-4ks.yml new file +10 -0
new file mode 100644
index 00000000..4c78e9a4
@@ -0,0 +1,10 @@
+---
+# XFS 32k block, 4k sector configuration
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 32768
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "32k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-4k-4ks-dev.yml new file +10 -0
new file mode 100644
index 00000000..f8b8c55b
@@ -0,0 +1,10 @@
+---
+# XFS 4k block, 4k sector configuration (dev)
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 4096
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "4k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml new file +10 -0
new file mode 100644
index 00000000..ffe9eb28
@@ -0,0 +1,10 @@
+---
+# XFS 4k block, 4k sector configuration
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 4096
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "4k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
\ No newline at end of file
playbooks/host_vars/debian13-ai-xfs-64k-4ks-dev.yml new file +10 -0
new file mode 100644
index 00000000..1590f154
@@ -0,0 +1,10 @@
+---
+# XFS 64k block, 4k sector configuration (dev)
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 65536
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "64k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/host_vars/debian13-ai-xfs-64k-4ks.yml new file +10 -0
new file mode 100644
index 00000000..482835c4
@@ -0,0 +1,10 @@
+---
+# XFS 64k block, 4k sector configuration
+ai_docker_fstype: "xfs"
+ai_docker_xfs_blocksize: 65536
+ai_docker_xfs_sectorsize: 4096
+ai_docker_xfs_mkfs_opts: ""
+filesystem_type: "xfs"
+filesystem_block_size: "64k-4ks"
+ai_filesystem: "xfs"
+ai_data_device_path: "/var/lib/docker"
playbooks/roles/ai_collect_results/files/analyze_results.py new file +979 -0

979 lines not on this page · raw message

playbooks/roles/ai_collect_results/files/generate_better_graphs.py new file +548 -0

548 lines not on this page · raw message

playbooks/roles/ai_collect_results/files/generate_graphs.py new file +678 -0

529 lines not on this page · raw message

playbooks/roles/ai_collect_results/files/generate_html_report.py new file +427 -0

427 lines not on this page · raw message

playbooks/roles/ai_collect_results/tasks/main.yml new file +220 -0

220 lines not on this page · raw message

playbooks/roles/ai_collect_results/templates/analysis_config.json.j2 new file +6 -0
new file mode 100644
index 00000000..5a879649

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_destroy/tasks/main.yml new file +63 -0

63 lines not on this page · raw message

playbooks/roles/ai_docker_storage/tasks/main.yml new file +123 -0

123 lines not on this page · raw message

playbooks/roles/ai_install/tasks/main.yml new file +90 -0

90 lines not on this page · raw message

playbooks/roles/ai_results/tasks/main.yml new file +22 -0
new file mode 100644
index 00000000..094a9025

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py new file +506 -0

506 lines not on this page · raw message

playbooks/roles/ai_run_benchmarks/tasks/main.yml new file +181 -0

181 lines not on this page · raw message

playbooks/roles/ai_run_benchmarks/templates/benchmark_config.json.j2 new file +24 -0
new file mode 100644
index 00000000..9983fc16

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_setup/tasks/main.yml new file +115 -0

115 lines not on this page · raw message

playbooks/roles/ai_uninstall/tasks/main.yml new file +62 -0

62 lines not on this page · raw message

playbooks/roles/gen_hosts/tasks/main.yml +14 -0
index ec11d039..4b35d9f6 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/hosts.j2 +108 -0

108 lines not on this page · raw message

playbooks/roles/gen_nodes/tasks/main.yml +34 -0
index a8598481..d54977be 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/milvus/README.md new file +181 -0

181 lines not on this page · raw message

playbooks/roles/milvus/defaults/main.yml new file +74 -0

74 lines not on this page · raw message

playbooks/roles/milvus/files/milvus_benchmark.py new file +348 -0

348 lines not on this page · raw message

playbooks/roles/milvus/files/milvus_utils.py new file +134 -0

134 lines not on this page · raw message

playbooks/roles/milvus/meta/main.yml new file +30 -0
new file mode 100644
index 00000000..6af514b7

Lines of this file are not shown. View the raw message.

playbooks/roles/milvus/tasks/benchmark.yml new file +61 -0

61 lines not on this page · raw message

playbooks/roles/milvus/tasks/benchmark_setup.yml new file +58 -0

58 lines not on this page · raw message

playbooks/roles/milvus/tasks/install_docker.yml new file +97 -0

97 lines not on this page · raw message

playbooks/roles/milvus/tasks/main.yml new file +52 -0

52 lines not on this page · raw message

playbooks/roles/milvus/tasks/setup.yml new file +107 -0

107 lines not on this page · raw message

playbooks/roles/milvus/templates/benchmark_config.json.j2 new file +25 -0
new file mode 100644
index 00000000..f3ed04a0

Lines of this file are not shown. View the raw message.

playbooks/roles/milvus/templates/docker-compose.override.yml.j2 new file +24 -0
new file mode 100644
index 00000000..b4f96a44

Lines of this file are not shown. View the raw message.

playbooks/roles/milvus/templates/docker-compose.yml.j2 new file +64 -0

64 lines not on this page · raw message

playbooks/roles/milvus/templates/milvus.yaml.j2 new file +30 -0
new file mode 100644
index 00000000..f843ec4b

Lines of this file are not shown. View the raw message.

playbooks/roles/milvus/templates/test_connection.py.j2 new file +25 -0
new file mode 100644
index 00000000..d85423ba

Lines of this file are not shown. View the raw message.

workflows/Makefile +4 -0
index b5f54ff5..fe35707b 100644

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig new file +164 -0

164 lines not on this page · raw message

workflows/ai/Kconfig.docker new file +172 -0

172 lines not on this page · raw message

workflows/ai/Kconfig.docker-storage new file +201 -0

201 lines not on this page · raw message

workflows/ai/Kconfig.native new file +184 -0

184 lines not on this page · raw message

workflows/ai/Makefile new file +160 -0

160 lines not on this page · raw message

workflows/ai/scripts/analysis_config.json new file +6 -0
new file mode 100644
index 00000000..2f90f4d5

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/analyze_results.py new file +979 -0

979 lines not on this page · raw message

workflows/ai/scripts/generate_graphs.py new file +1174 -0

1,174 lines not on this page · raw message

workflows/ai/scripts/generate_html_report.py new file +558 -0

558 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.50.1

Part of the diff is not shown here. View the raw message.

#2 lore raw

PATCH 2/2 ai: add multi-filesystem testing support for Milvus benchmarks

In reply to Luis Chamberlain

Extend the AI workflow to support testing Milvus across multiple filesystem configurations simultaneously. This enables comprehensive performance comparisons between different filesystems and their configuration options.

Key features:
- Dynamic node generation based on enabled filesystem configurations
- Support for XFS, EXT4, and BTRFS with various mount options
- Per-filesystem result collection and analysis
- A/B testing across all filesystem configurations
- Automated comparison graphs between filesystems

Filesystem configurations:
- XFS: default, nocrc, bigtime with various block sizes (512, 1k, 2k, 4k)
- EXT4: default, nojournal, bigalloc configurations
- BTRFS: default, zlib, lzo, zstd compression options

Defconfigs:
- ai-milvus-multifs: Test 7 filesystem configs with A/B testing
- ai-milvus-multifs-distro: Test with distribution kernels
- ai-milvus-multifs-extended: Extended configs (14 filesystems total)

Node generation:
The system dynamically generates nodes based on enabled filesystem configurations. With A/B testing enabled, this creates baseline and dev nodes for each filesystem (e.g., debian13-ai-xfs-4k and debian13-ai-xfs-4k-dev).

Usage:
make defconfig-ai-milvus-multifs

  make bringup    # Creates nodes for each filesystem
  make ai         # Setup infrastructure on all nodes
  make ai-tests   # Run benchmarks on all filesystems

make ai-results # Collect and compare results

This enables systematic evaluation of how different filesystems and their configurations affect vector database performance.

Generated-by Claude AI
---
 .github/workflows/docker-tests.yml            |    6 +
 Makefile                                      |    2 +-
 defconfigs/ai-milvus-multifs                  |   67 +
 defconfigs/ai-milvus-multifs-distro           |  109 ++
 defconfigs/ai-milvus-multifs-extended         |  108 ++
 docs/ai/vector-databases/README.md            |    1 -
 playbooks/ai_install.yml                      |    6 +
 playbooks/ai_multifs.yml                      |   24 +
 .../host_vars/debian13-ai-xfs-4k-4ks.yml      |   10 -
 .../files/analyze_results.py                  | 1132 +++++++++++---
 .../files/generate_better_graphs.py           |   16 +-
 .../files/generate_graphs.py                  |  888 ++++-------
 .../files/generate_html_report.py             |  263 +++-
 .../roles/ai_collect_results/tasks/main.yml   |   42 +-
 .../templates/analysis_config.json.j2         |    2 +-
 .../roles/ai_milvus_storage/tasks/main.yml    |  161 ++
 .../tasks/generate_comparison.yml             |  279 ++++
 playbooks/roles/ai_multifs_run/tasks/main.yml |   23 +
 .../tasks/run_single_filesystem.yml           |  104 ++
 .../templates/milvus_config.json.j2           |   42 +
 .../roles/ai_multifs_setup/defaults/main.yml  |   49 +
 .../roles/ai_multifs_setup/tasks/main.yml     |   70 +
 .../files/milvus_benchmark.py                 |  164 +-
 playbooks/roles/gen_hosts/tasks/main.yml      |   19 +
 .../roles/gen_hosts/templates/fstests.j2      |    2 +
 playbooks/roles/gen_hosts/templates/gitr.j2   |    2 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |   35 +-
 .../roles/gen_hosts/templates/nfstest.j2      |    2 +
 playbooks/roles/gen_hosts/templates/pynfs.j2  |    2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |   90 ++
 .../roles/guestfs/tasks/bringup/main.yml      |   15 +
 scripts/guestfs.Makefile                      |    2 +-
 workflows/ai/Kconfig                          |   13 +
 workflows/ai/Kconfig.fs                       |  118 ++
 workflows/ai/Kconfig.multifs                  |  184 +++
 workflows/ai/scripts/analysis_config.json     |    2 +-
 workflows/ai/scripts/analyze_results.py       | 1132 +++++++++++---
 workflows/ai/scripts/generate_graphs.py       | 1372 ++++-------------
 workflows/ai/scripts/generate_html_report.py  |   94 +-
 39 files changed, 4356 insertions(+), 2296 deletions(-)
 create mode 100644 defconfigs/ai-milvus-multifs
 create mode 100644 defconfigs/ai-milvus-multifs-distro
 create mode 100644 defconfigs/ai-milvus-multifs-extended
 create mode 100644 playbooks/ai_multifs.yml
 delete mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml
 create mode 100644 playbooks/roles/ai_milvus_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml
 create mode 100644 playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2
 create mode 100644 playbooks/roles/ai_multifs_setup/defaults/main.yml
 create mode 100644 playbooks/roles/ai_multifs_setup/tasks/main.yml
 create mode 100644 workflows/ai/Kconfig.fs
 create mode 100644 workflows/ai/Kconfig.multifs
.github/workflows/docker-tests.yml +6 -0
index c0e0d03d..adea1182 100644
@@ -53,3 +53,9 @@ jobs:
           echo "Running simple make targets on ${{ matrix.distro_container }} environment"
           make mrproper
 
+      - name: Test fio-tests defconfig
+        run: |
+          echo "Testing fio-tests CI configuration"
+          make defconfig-fio-tests-ci
+          make
+          echo "Configuration test passed for fio-tests"
Makefile +1 -1
index 8755577e..83c67340 100644
@@ -226,7 +226,7 @@ include scripts/bringup.Makefile
 endif
 
 DEFAULT_DEPS += $(ANSIBLE_INVENTORY_FILE)
-$(ANSIBLE_INVENTORY_FILE): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_HOSTS_TEMPLATE)
+$(ANSIBLE_INVENTORY_FILE): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_HOSTS_TEMPLATE) $(KDEVOPS_NODES)
 	$(Q)ANSIBLE_LOCALHOST_WARNING=False ANSIBLE_INVENTORY_UNPARSED_WARNING=False \
 		ansible-playbook $(ANSIBLE_VERBOSE) \
 		$(KDEVOPS_PLAYBOOKS_DIR)/gen_hosts.yml \
defconfigs/ai-milvus-multifs new file +67 -0

68 lines not on this page · raw message

defconfigs/ai-milvus-multifs-distro new file +109 -0

109 lines not on this page · raw message

defconfigs/ai-milvus-multifs-extended new file +108 -0

108 lines not on this page · raw message

docs/ai/vector-databases/README.md +0 -1
index 2a3955d7..0fdd204b 100644
@@ -52,7 +52,6 @@ Vector databases heavily depend on storage performance. The workflow tests acros
 - **XFS**: Default for many production deployments
 - **ext4**: Traditional Linux filesystem
 - **btrfs**: Copy-on-write with compression support
-- **ZFS**: Advanced features for data integrity
 
 ## Configuration Dimensions
 
playbooks/ai_install.yml +6 -0
index 70b734e4..38e6671c 100644
@@ -4,5 +4,11 @@
   become: true
   become_user: root
   roles:
+    - role: ai_docker_storage
+      when: ai_docker_storage_enable | default(true)
+      tags: ['ai', 'docker', 'storage']
+    - role: ai_milvus_storage
+      when: ai_milvus_storage_enable | default(false)
+      tags: ['ai', 'milvus', 'storage']
     - role: milvus
       tags: ['ai', 'vector_db', 'milvus', 'install']
playbooks/ai_multifs.yml new file +24 -0
new file mode 100644
index 00000000..637f11f4
@@ -0,0 +1,24 @@
+---
+- hosts: baseline
+  become: yes
+  gather_facts: yes
+  vars:
+    ai_benchmark_results_dir: "{{ ai_multifs_results_dir | default('/data/ai-multifs-benchmark') }}"
+  roles:
+    - role: ai_multifs_setup
+    - role: ai_multifs_run
+  tasks:
+    - name: Final multi-filesystem testing summary
+      debug:
+        msg: |
+          Multi-filesystem AI benchmark testing completed!
+
+          Results directory: {{ ai_multifs_results_dir }}
+          Comparison report: {{ ai_multifs_results_dir }}/comparison/multi_filesystem_comparison.html
+
+          Individual filesystem results:
+          {% for config in ai_multifs_configurations %}
+          {% if config.enabled %}
+          - {{ config.name }}: {{ ai_multifs_results_dir }}/{{ config.name }}/
+          {% endif %}
+          {% endfor %}
playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml deleted +0 -10
deleted file mode 100644
index ffe9eb28..00000000
@@ -1,10 +0,0 @@
----
-# XFS 4k block, 4k sector configuration
-ai_docker_fstype: "xfs"
-ai_docker_xfs_blocksize: 4096
-ai_docker_xfs_sectorsize: 4096
-ai_docker_xfs_mkfs_opts: ""
-filesystem_type: "xfs"
-filesystem_block_size: "4k-4ks"
-ai_filesystem: "xfs"
-ai_data_device_path: "/var/lib/docker"
\ No newline at end of file
playbooks/roles/ai_collect_results/files/analyze_results.py +927 -205

1,258 lines not on this page · raw message

playbooks/roles/ai_collect_results/files/generate_better_graphs.py +9 -7
index 645bac9e..b3681ff9 100755
@@ -29,17 +29,18 @@ def extract_filesystem_from_filename(filename):
         if "_" in node_name:
             parts = node_name.split("_")
             node_name = "_".join(parts[:-1])  # Remove last part (iteration)
-        
+
         # Extract filesystem type from node name
         if "-xfs-" in node_name:
             return "xfs"
         elif "-ext4-" in node_name:
-            return "ext4"  
+            return "ext4"
         elif "-btrfs-" in node_name:
             return "btrfs"
-    
+
     return "unknown"
 
+
 def extract_node_config_from_filename(filename):
     """Extract detailed node configuration from filename"""
     # Expected format: results_debian13-ai-xfs-4k-4ks_1.json
@@ -50,14 +51,15 @@ def extract_node_config_from_filename(filename):
         if "_" in node_name:
             parts = node_name.split("_")
             node_name = "_".join(parts[:-1])  # Remove last part (iteration)
-        
+
         # Remove -dev suffix if present
         node_name = node_name.replace("-dev", "")
-        
+
         return node_name.replace("debian13-ai-", "")
-    
+
     return "unknown"
 
+
 def detect_filesystem():
     """Detect the filesystem type of /data on test nodes"""
     # This is now a fallback - we primarily use filename-based detection
@@ -104,7 +106,7 @@ def load_results(results_dir):
                 # Extract node type from filename
                 filename = os.path.basename(json_file)
                 data["filename"] = filename
-                
+
                 # Extract filesystem type and config from filename
                 data["filesystem"] = extract_filesystem_from_filename(filename)
                 data["node_config"] = extract_node_config_from_filename(filename)
playbooks/roles/ai_collect_results/files/generate_graphs.py +286 -602

953 lines not on this page · raw message

playbooks/roles/ai_collect_results/files/generate_html_report.py +223 -40

370 lines not on this page · raw message

playbooks/roles/ai_collect_results/tasks/main.yml +12 -30

17 lines not on this page · raw message

playbooks/roles/ai_collect_results/templates/analysis_config.json.j2 +1 -1
index 5a879649..459cd602 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_milvus_storage/tasks/main.yml new file +161 -0

161 lines not on this page · raw message

playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml new file +279 -0

279 lines not on this page · raw message

playbooks/roles/ai_multifs_run/tasks/main.yml new file +23 -0
new file mode 100644
index 00000000..38dbba12

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml new file +104 -0

104 lines not on this page · raw message

playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2 new file +42 -0

42 lines not on this page · raw message

playbooks/roles/ai_multifs_setup/defaults/main.yml new file +49 -0

49 lines not on this page · raw message

playbooks/roles/ai_multifs_setup/tasks/main.yml new file +70 -0

70 lines not on this page · raw message

playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py +107 -57

164 lines not on this page · raw message

playbooks/roles/gen_hosts/tasks/main.yml +19 -0
index 4b35d9f6..d36790b0 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/fstests.j2 +2 -0
index ac086c6e..32d90abf 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/gitr.j2 +2 -0
index 7f9094d4..3f30a5fb 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/hosts.j2 +13 -22
index cdcd1883..e9441605 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/nfstest.j2 +2 -0
index e427ac34..709d871d 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/pynfs.j2 +2 -0
index 85c87dae..55add4d1 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_nodes/tasks/main.yml +90 -0

90 lines not on this page · raw message

playbooks/roles/guestfs/tasks/bringup/main.yml +15 -0
index c131de25..bd9f5260 100644

Lines of this file are not shown. View the raw message.

scripts/guestfs.Makefile +1 -1
index bd03f58c..f6c350a4 100644

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig +13 -0
index 2ffc6b65..d04570d8 100644

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig.fs new file +118 -0

118 lines not on this page · raw message

workflows/ai/Kconfig.multifs new file +184 -0

184 lines not on this page · raw message

workflows/ai/scripts/analysis_config.json +1 -1
index 2f90f4d5..5f0a9328 100644

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/analyze_results.py +927 -205

1,132 lines not on this page · raw message

workflows/ai/scripts/generate_graphs.py +280 -1092

1,372 lines not on this page · raw message

workflows/ai/scripts/generate_html_report.py +73 -21

94 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.50.1

Part of the diff is not shown here. View the raw message.

#3 lore raw

Re: [PATCH 2/2] ai: add multi-filesystem testing support for Milvus benchmarks

Quoted 33 lines · Luis Chamberlain

Extend the AI workflow to support testing Milvus across multiple filesystem configurations simultaneously. This enables comprehensive performance comparisons between different filesystems and their configuration options.

Key features:
- Dynamic node generation based on enabled filesystem configurations
- Support for XFS, EXT4, and BTRFS with various mount options
- Per-filesystem result collection and analysis
- A/B testing across all filesystem configurations
- Automated comparison graphs between filesystems

Filesystem configurations:
- XFS: default, nocrc, bigtime with various block sizes (512, 1k, 2k, 4k)
- EXT4: default, nojournal, bigalloc configurations
- BTRFS: default, zlib, lzo, zstd compression options

Defconfigs:
- ai-milvus-multifs: Test 7 filesystem configs with A/B testing
- ai-milvus-multifs-distro: Test with distribution kernels
- ai-milvus-multifs-extended: Extended configs (14 filesystems total)

Node generation:
The system dynamically generates nodes based on enabled filesystem configurations. With A/B testing enabled, this creates baseline and dev nodes for each filesystem (e.g., debian13-ai-xfs-4k and debian13-ai-xfs-4k-dev).

Usage:
make defconfig-ai-milvus-multifs

  make bringup    # Creates nodes for each filesystem
  make ai         # Setup infrastructure on all nodes
  make ai-tests   # Run benchmarks on all filesystems
Quoted 7 lines

make ai-results # Collect and compare results

This enables systematic evaluation of how different filesystems and their configurations affect vector database performance.

Generated-by Claude AI

Hey Luis -

I'm looking at adding "AI optimized" and "GPU optimized" machine size choices in the cloud provider Kconfigs. I assume this set can take advantage of those. Any suggestions, or let me know if I'm way off base.

Quoted 7,753 lines
---
 .github/workflows/docker-tests.yml            |    6 +
 Makefile                                      |    2 +-
 defconfigs/ai-milvus-multifs                  |   67 +
 defconfigs/ai-milvus-multifs-distro           |  109 ++
 defconfigs/ai-milvus-multifs-extended         |  108 ++
 docs/ai/vector-databases/README.md            |    1 -
 playbooks/ai_install.yml                      |    6 +
 playbooks/ai_multifs.yml                      |   24 +
 .../host_vars/debian13-ai-xfs-4k-4ks.yml      |   10 -
 .../files/analyze_results.py                  | 1132 +++++++++++---
 .../files/generate_better_graphs.py           |   16 +-
 .../files/generate_graphs.py                  |  888 ++++-------
 .../files/generate_html_report.py             |  263 +++-
 .../roles/ai_collect_results/tasks/main.yml   |   42 +-
 .../templates/analysis_config.json.j2         |    2 +-
 .../roles/ai_milvus_storage/tasks/main.yml    |  161 ++
 .../tasks/generate_comparison.yml             |  279 ++++
 playbooks/roles/ai_multifs_run/tasks/main.yml |   23 +
 .../tasks/run_single_filesystem.yml           |  104 ++
 .../templates/milvus_config.json.j2           |   42 +
 .../roles/ai_multifs_setup/defaults/main.yml  |   49 +
 .../roles/ai_multifs_setup/tasks/main.yml     |   70 +
 .../files/milvus_benchmark.py                 |  164 +-
 playbooks/roles/gen_hosts/tasks/main.yml      |   19 +
 .../roles/gen_hosts/templates/fstests.j2      |    2 +
 playbooks/roles/gen_hosts/templates/gitr.j2   |    2 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |   35 +-
 .../roles/gen_hosts/templates/nfstest.j2      |    2 +
 playbooks/roles/gen_hosts/templates/pynfs.j2  |    2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |   90 ++
 .../roles/guestfs/tasks/bringup/main.yml      |   15 +
 scripts/guestfs.Makefile                      |    2 +-
 workflows/ai/Kconfig                          |   13 +
 workflows/ai/Kconfig.fs                       |  118 ++
 workflows/ai/Kconfig.multifs                  |  184 +++
 workflows/ai/scripts/analysis_config.json     |    2 +-
 workflows/ai/scripts/analyze_results.py       | 1132 +++++++++++---
 workflows/ai/scripts/generate_graphs.py       | 1372 ++++-------------
 workflows/ai/scripts/generate_html_report.py  |   94 +-
 39 files changed, 4356 insertions(+), 2296 deletions(-)
 create mode 100644 defconfigs/ai-milvus-multifs
 create mode 100644 defconfigs/ai-milvus-multifs-distro
 create mode 100644 defconfigs/ai-milvus-multifs-extended
 create mode 100644 playbooks/ai_multifs.yml
 delete mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml
 create mode 100644 playbooks/roles/ai_milvus_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml
 create mode 100644 playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2
 create mode 100644 playbooks/roles/ai_multifs_setup/defaults/main.yml
 create mode 100644 playbooks/roles/ai_multifs_setup/tasks/main.yml
 create mode 100644 workflows/ai/Kconfig.fs
 create mode 100644 workflows/ai/Kconfig.multifs
.github/workflows/docker-tests.yml +6 -0
index c0e0d03d..adea1182 100644
@@ -53,3 +53,9 @@ jobs:
           echo "Running simple make targets on ${{ matrix.distro_container }} environment"
           make mrproper
 
+      - name: Test fio-tests defconfig
+        run: |
+          echo "Testing fio-tests CI configuration"
+          make defconfig-fio-tests-ci
+          make
+          echo "Configuration test passed for fio-tests"
Makefile +1 -1
index 8755577e..83c67340 100644
@@ -226,7 +226,7 @@ include scripts/bringup.Makefile
 endif
 
 DEFAULT_DEPS += $(ANSIBLE_INVENTORY_FILE)
-$(ANSIBLE_INVENTORY_FILE): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_HOSTS_TEMPLATE)
+$(ANSIBLE_INVENTORY_FILE): .config $(ANSIBLE_CFG_FILE) $(KDEVOPS_HOSTS_TEMPLATE) $(KDEVOPS_NODES)
 	$(Q)ANSIBLE_LOCALHOST_WARNING=False ANSIBLE_INVENTORY_UNPARSED_WARNING=False \
 		ansible-playbook $(ANSIBLE_VERBOSE) \
 		$(KDEVOPS_PLAYBOOKS_DIR)/gen_hosts.yml \
defconfigs/ai-milvus-multifs new file +67 -0
new file mode 100644
index 00000000..7e5ad971
@@ -0,0 +1,67 @@
+CONFIG_GUESTFS=y
+CONFIG_LIBVIRT=y
+
+# Disable mirror features for CI/testing
+# CONFIG_ENABLE_LOCAL_LINUX_MIRROR is not set
+# CONFIG_USE_LOCAL_LINUX_MIRROR is not set
+# CONFIG_INSTALL_ONLY_GIT_DAEMON is not set
+# CONFIG_MIRROR_INSTALL is not set
+
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+
+CONFIG_BOOTLINUX=y
+CONFIG_BOOTLINUX_9P=y
+
+# Enable A/B testing with different kernel references
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y
+CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y
+
+# AI workflow configuration
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
+
+# Vector database configuration
+CONFIG_AI_TESTS_VECTOR_DATABASE=y
+CONFIG_AI_VECTOR_DB_MILVUS=y
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
+
+# Enable multi-filesystem testing
+CONFIG_AI_MULTIFS_ENABLE=y
+CONFIG_AI_ENABLE_MULTIFS_TESTING=y
+
+# Enable dedicated Milvus storage with node-based filesystem
+CONFIG_AI_MILVUS_STORAGE_ENABLE=y
+CONFIG_AI_MILVUS_USE_NODE_FS=y
+
+# Test XFS with different block sizes
+CONFIG_AI_MULTIFS_TEST_XFS=y
+CONFIG_AI_MULTIFS_XFS_4K_4KS=y
+CONFIG_AI_MULTIFS_XFS_16K_4KS=y
+CONFIG_AI_MULTIFS_XFS_32K_4KS=y
+CONFIG_AI_MULTIFS_XFS_64K_4KS=y
+
+# Test EXT4 configurations
+CONFIG_AI_MULTIFS_TEST_EXT4=y
+CONFIG_AI_MULTIFS_EXT4_4K=y
+CONFIG_AI_MULTIFS_EXT4_16K_BIGALLOC=y
+
+# Test BTRFS
+CONFIG_AI_MULTIFS_TEST_BTRFS=y
+CONFIG_AI_MULTIFS_BTRFS_DEFAULT=y
+
+# Performance settings
+CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
+CONFIG_AI_BENCHMARK_ITERATIONS=5
+
+# Dataset configuration for benchmarking
+CONFIG_AI_VECTOR_DB_MILVUS_DATASET_SIZE=100000
+CONFIG_AI_VECTOR_DB_MILVUS_BATCH_SIZE=10000
+CONFIG_AI_VECTOR_DB_MILVUS_NUM_QUERIES=10000
+
+# Container configuration
+CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_IMAGE_2_5=y
+CONFIG_AI_VECTOR_DB_MILVUS_MEMORY_LIMIT="8g"
+CONFIG_AI_VECTOR_DB_MILVUS_CPU_LIMIT="4.0"
\ No newline at end of file
defconfigs/ai-milvus-multifs-distro new file +109 -0
new file mode 100644
index 00000000..fb71f2b5
@@ -0,0 +1,109 @@
+# AI Multi-Filesystem Performance Testing Configuration (Distro Kernel)
+# This configuration enables testing AI workloads across multiple filesystem
+# configurations including XFS (4k and 16k block sizes), ext4 (4k and 16k bigalloc),
+# and btrfs (default profile) using the distribution kernel without A/B testing.
+
+# Base virtualization setup
+CONFIG_LIBVIRT=y
+CONFIG_LIBVIRT_MACHINE_TYPE_Q35=y
+CONFIG_LIBVIRT_STORAGE_POOL_PATH="/opt/kdevops/libvirt"
+CONFIG_LIBVIRT_ENABLE_LARGEIO=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE="50GiB"
+
+# Network configuration
+CONFIG_LIBVIRT_ENABLE_BRIDGED_NETWORKING=y
+CONFIG_LIBVIRT_NET_NAME="kdevops"
+
+# Host configuration
+CONFIG_KDEVOPS_HOSTS_TEMPLATE="hosts.j2"
+CONFIG_VAGRANT_NVME_DISK_SIZE="50GiB"
+
+# Base system requirements
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
+
+# AI Workflow Configuration
+CONFIG_KDEVOPS_WORKFLOW_ENABLE_AI=y
+CONFIG_AI_TESTS_VECTOR_DATABASE=y
+CONFIG_AI_MILVUS_DOCKER=y
+CONFIG_AI_VECTOR_DB_TYPE_MILVUS=y
+
+# Milvus Configuration
+CONFIG_AI_MILVUS_HOST="localhost"
+CONFIG_AI_MILVUS_PORT=19530
+CONFIG_AI_MILVUS_DATABASE_NAME="ai_benchmark"
+
+# Test Parameters (optimized for multi-fs testing)
+CONFIG_AI_BENCHMARK_ITERATIONS=3
+CONFIG_AI_DATASET_1M=y
+CONFIG_AI_VECTOR_DIM_128=y
+CONFIG_AI_BENCHMARK_RUNTIME="180"
+CONFIG_AI_BENCHMARK_WARMUP_TIME="30"
+
+# Query patterns
+CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
+CONFIG_AI_BENCHMARK_QUERY_TOPK_10=y
+
+# Batch sizes
+CONFIG_AI_BENCHMARK_BATCH_1=y
+CONFIG_AI_BENCHMARK_BATCH_10=y
+
+# Index configuration
+CONFIG_AI_INDEX_HNSW=y
+CONFIG_AI_INDEX_TYPE="HNSW"
+CONFIG_AI_INDEX_HNSW_M=16
+CONFIG_AI_INDEX_HNSW_EF_CONSTRUCTION=200
+CONFIG_AI_INDEX_HNSW_EF=64
+
+# Results and visualization
+CONFIG_AI_BENCHMARK_RESULTS_DIR="/data/ai-benchmark"
+CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
+CONFIG_AI_BENCHMARK_GRAPH_FORMAT="png"
+CONFIG_AI_BENCHMARK_GRAPH_DPI=300
+CONFIG_AI_BENCHMARK_GRAPH_THEME="default"
+
+# Multi-filesystem testing configuration
+CONFIG_AI_ENABLE_MULTIFS_TESTING=y
+CONFIG_AI_MULTIFS_RESULTS_DIR="/data/ai-multifs-benchmark"
+
+# Enable dedicated Milvus storage with node-based filesystem
+CONFIG_AI_MILVUS_STORAGE_ENABLE=y
+CONFIG_AI_MILVUS_USE_NODE_FS=y
+CONFIG_AI_MILVUS_DEVICE="/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops3"
+CONFIG_AI_MILVUS_MOUNT_POINT="/data/milvus"
+
+# XFS configurations
+CONFIG_AI_MULTIFS_TEST_XFS=y
+CONFIG_AI_MULTIFS_XFS_4K_4KS=y
+CONFIG_AI_MULTIFS_XFS_16K_4KS=y
+CONFIG_AI_MULTIFS_XFS_32K_4KS=y
+CONFIG_AI_MULTIFS_XFS_64K_4KS=y
+
+# ext4 configurations
+CONFIG_AI_MULTIFS_TEST_EXT4=y
+CONFIG_AI_MULTIFS_EXT4_4K=y
+CONFIG_AI_MULTIFS_EXT4_16K_BIGALLOC=y
+
+# btrfs configurations
+CONFIG_AI_MULTIFS_TEST_BTRFS=y
+CONFIG_AI_MULTIFS_BTRFS_DEFAULT=y
+
+# Standard filesystem configuration (for comparison)
+CONFIG_AI_FILESYSTEM_XFS=y
+CONFIG_AI_FILESYSTEM="xfs"
+CONFIG_AI_FSTYPE="xfs"
+CONFIG_AI_XFS_MKFS_OPTS="-f -s size=4096"
+CONFIG_AI_XFS_MOUNT_OPTS="rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota"
+
+# Use distribution kernel (no kernel building)
+# CONFIG_BOOTLINUX is not set
+
+# Memory configuration
+CONFIG_LIBVIRT_MEM_MB=16384
+
+# Disable A/B testing to use single baseline configuration
+# CONFIG_KDEVOPS_BASELINE_AND_DEV is not set
defconfigs/ai-milvus-multifs-extended new file +108 -0
new file mode 100644
index 00000000..7886c8c4
@@ -0,0 +1,108 @@
+# AI Extended Multi-Filesystem Performance Testing Configuration (Distro Kernel)
+# This configuration enables testing AI workloads across multiple filesystem
+# configurations including XFS (4k, 16k, 32k, 64k block sizes), ext4 (4k and 16k bigalloc),
+# and btrfs (default profile) using the distribution kernel without A/B testing.
+
+# Base virtualization setup
+CONFIG_LIBVIRT=y
+CONFIG_LIBVIRT_MACHINE_TYPE_Q35=y
+CONFIG_LIBVIRT_STORAGE_POOL_PATH="/opt/kdevops/libvirt"
+CONFIG_LIBVIRT_ENABLE_LARGEIO=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE="50GiB"
+
+# Network configuration
+CONFIG_LIBVIRT_ENABLE_BRIDGED_NETWORKING=y
+CONFIG_LIBVIRT_NET_NAME="kdevops"
+
+# Host configuration
+CONFIG_KDEVOPS_HOSTS_TEMPLATE="hosts.j2"
+CONFIG_VAGRANT_NVME_DISK_SIZE="50GiB"
+
+# Base system requirements
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
+
+# AI Workflow Configuration
+CONFIG_KDEVOPS_WORKFLOW_ENABLE_AI=y
+CONFIG_AI_TESTS_VECTOR_DATABASE=y
+CONFIG_AI_VECTOR_DB_MILVUS=y
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
+
+# Test Parameters (optimized for multi-fs testing)
+CONFIG_AI_BENCHMARK_ITERATIONS=3
+CONFIG_AI_DATASET_1M=y
+CONFIG_AI_VECTOR_DIM_128=y
+CONFIG_AI_BENCHMARK_RUNTIME="180"
+CONFIG_AI_BENCHMARK_WARMUP_TIME="30"
+
+# Query patterns
+CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
+CONFIG_AI_BENCHMARK_QUERY_TOPK_10=y
+
+# Batch sizes
+CONFIG_AI_BENCHMARK_BATCH_1=y
+CONFIG_AI_BENCHMARK_BATCH_10=y
+
+# Index configuration
+CONFIG_AI_INDEX_HNSW=y
+CONFIG_AI_INDEX_TYPE="HNSW"
+CONFIG_AI_INDEX_HNSW_M=16
+CONFIG_AI_INDEX_HNSW_EF_CONSTRUCTION=200
+CONFIG_AI_INDEX_HNSW_EF=64
+
+# Results and visualization
+CONFIG_AI_BENCHMARK_RESULTS_DIR="/data/ai-benchmark"
+CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
+CONFIG_AI_BENCHMARK_GRAPH_FORMAT="png"
+CONFIG_AI_BENCHMARK_GRAPH_DPI=300
+CONFIG_AI_BENCHMARK_GRAPH_THEME="default"
+
+# Multi-filesystem testing configuration
+CONFIG_AI_MULTIFS_ENABLE=y
+CONFIG_AI_ENABLE_MULTIFS_TESTING=y
+CONFIG_AI_MULTIFS_RESULTS_DIR="/data/ai-multifs-benchmark"
+
+# Enable dedicated Milvus storage with node-based filesystem
+CONFIG_AI_MILVUS_STORAGE_ENABLE=y
+CONFIG_AI_MILVUS_USE_NODE_FS=y
+CONFIG_AI_MILVUS_DEVICE="/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops3"
+CONFIG_AI_MILVUS_MOUNT_POINT="/data/milvus"
+
+# Extended XFS configurations (4k, 16k, 32k, 64k block sizes)
+CONFIG_AI_MULTIFS_TEST_XFS=y
+CONFIG_AI_MULTIFS_XFS_4K_4KS=y
+CONFIG_AI_MULTIFS_XFS_16K_4KS=y
+CONFIG_AI_MULTIFS_XFS_32K_4KS=y
+CONFIG_AI_MULTIFS_XFS_64K_4KS=y
+
+# ext4 configurations
+CONFIG_AI_MULTIFS_TEST_EXT4=y
+CONFIG_AI_MULTIFS_EXT4_4K=y
+CONFIG_AI_MULTIFS_EXT4_16K_BIGALLOC=y
+
+# btrfs configurations
+CONFIG_AI_MULTIFS_TEST_BTRFS=y
+CONFIG_AI_MULTIFS_BTRFS_DEFAULT=y
+
+# Standard filesystem configuration (for comparison)
+CONFIG_AI_FILESYSTEM_XFS=y
+CONFIG_AI_FILESYSTEM="xfs"
+CONFIG_AI_FSTYPE="xfs"
+CONFIG_AI_XFS_MKFS_OPTS="-f -s size=4096"
+CONFIG_AI_XFS_MOUNT_OPTS="rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota"
+
+# Use distribution kernel (no kernel building)
+# CONFIG_BOOTLINUX is not set
+
+# Memory configuration
+CONFIG_LIBVIRT_MEM_MB=16384
+
+# Baseline/dev testing setup
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y
+# Build Linux
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y
docs/ai/vector-databases/README.md +0 -1
index 2a3955d7..0fdd204b 100644
@@ -52,7 +52,6 @@ Vector databases heavily depend on storage performance. The workflow tests acros
 - **XFS**: Default for many production deployments
 - **ext4**: Traditional Linux filesystem
 - **btrfs**: Copy-on-write with compression support
-- **ZFS**: Advanced features for data integrity
 
 ## Configuration Dimensions
 
playbooks/ai_install.yml +6 -0
index 70b734e4..38e6671c 100644
@@ -4,5 +4,11 @@
   become: true
   become_user: root
   roles:
+    - role: ai_docker_storage
+      when: ai_docker_storage_enable | default(true)
+      tags: ['ai', 'docker', 'storage']
+    - role: ai_milvus_storage
+      when: ai_milvus_storage_enable | default(false)
+      tags: ['ai', 'milvus', 'storage']
     - role: milvus
       tags: ['ai', 'vector_db', 'milvus', 'install']
playbooks/ai_multifs.yml new file +24 -0
new file mode 100644
index 00000000..637f11f4
@@ -0,0 +1,24 @@
+---
+- hosts: baseline
+  become: yes
+  gather_facts: yes
+  vars:
+    ai_benchmark_results_dir: "{{ ai_multifs_results_dir | default('/data/ai-multifs-benchmark') }}"
+  roles:
+    - role: ai_multifs_setup
+    - role: ai_multifs_run
+  tasks:
+    - name: Final multi-filesystem testing summary
+      debug:
+        msg: |
+          Multi-filesystem AI benchmark testing completed!
+
+          Results directory: {{ ai_multifs_results_dir }}
+          Comparison report: {{ ai_multifs_results_dir }}/comparison/multi_filesystem_comparison.html
+
+          Individual filesystem results:
+          {% for config in ai_multifs_configurations %}
+          {% if config.enabled %}
+          - {{ config.name }}: {{ ai_multifs_results_dir }}/{{ config.name }}/
+          {% endif %}
+          {% endfor %}
playbooks/host_vars/debian13-ai-xfs-4k-4ks.yml deleted +0 -10
deleted file mode 100644
index ffe9eb28..00000000
@@ -1,10 +0,0 @@
----
-# XFS 4k block, 4k sector configuration
-ai_docker_fstype: "xfs"
-ai_docker_xfs_blocksize: 4096
-ai_docker_xfs_sectorsize: 4096
-ai_docker_xfs_mkfs_opts: ""
-filesystem_type: "xfs"
-filesystem_block_size: "4k-4ks"
-ai_filesystem: "xfs"
-ai_data_device_path: "/var/lib/docker"
\ No newline at end of file
playbooks/roles/ai_collect_results/files/analyze_results.py +927 -205
index 3d11fb11..2dc4a1d6 100755
@@ -226,6 +226,68 @@ class ResultsAnalyzer:
 
         return fs_info
 
+    def _extract_filesystem_config(
+        self, result: Dict[str, Any]
+    ) -> tuple[str, str, str]:
+        """Extract filesystem type and block size from result data.
+        Returns (fs_type, block_size, config_key)"""
+        filename = result.get("_file", "")
+
+        # Primary: Extract filesystem type from filename (more reliable than JSON)
+        fs_type = "unknown"
+        block_size = "default"
+
+        if "xfs" in filename:
+            fs_type = "xfs"
+            # Check larger sizes first to avoid substring matches
+            if "64k" in filename and "64k-" in filename:
+                block_size = "64k"
+            elif "32k" in filename and "32k-" in filename:
+                block_size = "32k"
+            elif "16k" in filename and "16k-" in filename:
+                block_size = "16k"
+            elif "4k" in filename and "4k-" in filename:
+                block_size = "4k"
+        elif "ext4" in filename:
+            fs_type = "ext4"
+            if "16k" in filename:
+                block_size = "16k"
+            elif "4k" in filename:
+                block_size = "4k"
+        elif "btrfs" in filename:
+            fs_type = "btrfs"
+            block_size = "default"
+        else:
+            # Fallback to JSON data if filename parsing fails
+            fs_type = result.get("filesystem", "unknown")
+            self.logger.warning(
+                f"Could not determine filesystem from filename {filename}, using JSON data: {fs_type}"
+            )
+
+        config_key = f"{fs_type}-{block_size}" if block_size != "default" else fs_type
+        return fs_type, block_size, config_key
+
+    def _extract_node_info(self, result: Dict[str, Any]) -> tuple[str, bool]:
+        """Extract node hostname and determine if it's a dev node.
+        Returns (hostname, is_dev_node)"""
+        # Get hostname from system_info (preferred) or fall back to filename
+        system_info = result.get("system_info", {})
+        hostname = system_info.get("hostname", "")
+
+        # If no hostname in system_info, try extracting from filename
+        if not hostname:
+            filename = result.get("_file", "")
+            # Remove results_ prefix and .json suffix
+            hostname = filename.replace("results_", "").replace(".json", "")
+            # Remove iteration number if present (_1, _2, etc.)
+            if "_" in hostname and hostname.split("_")[-1].isdigit():
+                hostname = "_".join(hostname.split("_")[:-1])
+
+        # Determine if this is a dev node
+        is_dev = hostname.endswith("-dev")
+
+        return hostname, is_dev
+
     def load_results(self) -> bool:
         """Load all result files from the results directory"""
         try:
@@ -391,6 +453,8 @@ class ResultsAnalyzer:
             html.append(
                 "        .highlight { background-color: #fff3cd; padding: 10px; border-radius: 3px; }"
             )
+            html.append("        .baseline-row { background-color: #e8f5e9; }")
+            html.append("        .dev-row { background-color: #e3f2fd; }")
             html.append("    </style>")
             html.append("</head>")
             html.append("<body>")
@@ -486,26 +550,69 @@ class ResultsAnalyzer:
             else:
                 html.append("        <p>No storage device information available.</p>")
 
-            # Filesystem section
-            html.append("        <h3>🗂️ Filesystem Configuration</h3>")
-            fs_info = self.system_info.get("filesystem_info", {})
-            html.append("        <table class='config-table'>")
-            html.append(
-                "            <tr><td>Filesystem Type</td><td>"
-                + str(fs_info.get("filesystem_type", "Unknown"))
-                + "</td></tr>"
-            )
-            html.append(
-                "            <tr><td>Mount Point</td><td>"
-                + str(fs_info.get("mount_point", "Unknown"))
-                + "</td></tr>"
-            )
-            html.append(
-                "            <tr><td>Mount Options</td><td>"
-                + str(fs_info.get("mount_options", "Unknown"))
-                + "</td></tr>"
-            )
-            html.append("        </table>")
+            # Node Configuration section - Extract from actual benchmark results
+            html.append("        <h3>🗂️ Node Configuration</h3>")
+
+            # Collect node and filesystem information from benchmark results
+            node_configs = {}
+            for result in self.results_data:
+                # Extract node information
+                hostname, is_dev = self._extract_node_info(result)
+                fs_type, block_size, config_key = self._extract_filesystem_config(
+                    result
+                )
+
+                system_info = result.get("system_info", {})
+                data_path = system_info.get("data_path", "/data/milvus")
+                mount_point = system_info.get("mount_point", "/data")
+                kernel_version = system_info.get("kernel_version", "unknown")
+
+                if hostname not in node_configs:
+                    node_configs[hostname] = {
+                        "hostname": hostname,
+                        "node_type": "Development" if is_dev else "Baseline",
+                        "filesystem": fs_type,
+                        "block_size": block_size,
+                        "data_path": data_path,
+                        "mount_point": mount_point,
+                        "kernel": kernel_version,
+                        "test_count": 0,
+                    }
+                node_configs[hostname]["test_count"] += 1
+
+            if node_configs:
+                html.append("        <table class='config-table'>")
+                html.append(
+                    "            <tr><th>Node</th><th>Type</th><th>Filesystem</th><th>Block Size</th><th>Data Path</th><th>Mount Point</th><th>Kernel</th><th>Tests</th></tr>"
+                )
+                # Sort nodes with baseline first, then dev
+                sorted_nodes = sorted(
+                    node_configs.items(),
+                    key=lambda x: (x[1]["node_type"] != "Baseline", x[0]),
+                )
+                for hostname, config_info in sorted_nodes:
+                    row_class = (
+                        "dev-row"
+                        if config_info["node_type"] == "Development"
+                        else "baseline-row"
+                    )
+                    html.append(f"            <tr class='{row_class}'>")
+                    html.append(f"                <td><strong>{hostname}</strong></td>")
+                    html.append(f"                <td>{config_info['node_type']}</td>")
+                    html.append(f"                <td>{config_info['filesystem']}</td>")
+                    html.append(f"                <td>{config_info['block_size']}</td>")
+                    html.append(f"                <td>{config_info['data_path']}</td>")
+                    html.append(
+                        f"                <td>{config_info['mount_point']}</td>"
+                    )
+                    html.append(f"                <td>{config_info['kernel']}</td>")
+                    html.append(f"                <td>{config_info['test_count']}</td>")
+                    html.append(f"            </tr>")
+                html.append("        </table>")
+            else:
+                html.append(
+                    "        <p>No node configuration data found in results.</p>"
+                )
             html.append("    </div>")
 
             # Test Configuration Section
@@ -551,92 +658,192 @@ class ResultsAnalyzer:
                 html.append("        </table>")
                 html.append("    </div>")
 
-            # Performance Results Section
+            # Performance Results Section - Per Node
             html.append("    <div class='section'>")
-            html.append("        <h2>📊 Performance Results Summary</h2>")
+            html.append("        <h2>📊 Performance Results by Node</h2>")
 
             if self.results_data:
-                # Insert performance
-                insert_times = [
-                    r.get("insert_performance", {}).get("total_time_seconds", 0)
-                    for r in self.results_data
-                ]
-                insert_rates = [
-                    r.get("insert_performance", {}).get("vectors_per_second", 0)
-                    for r in self.results_data
-                ]
-
-                if insert_times and any(t > 0 for t in insert_times):
-                    html.append("        <h3>📈 Vector Insert Performance</h3>")
-                    html.append("        <table class='metric-table'>")
-                    html.append(
-                        f"            <tr><td>Average Insert Time</td><td>{np.mean(insert_times):.2f} seconds</td></tr>"
-                    )
-                    html.append(
-                        f"            <tr><td>Average Insert Rate</td><td>{np.mean(insert_rates):.2f} vectors/sec</td></tr>"
+                # Group results by node
+                node_performance = {}
+
+                for result in self.results_data:
+                    # Use node hostname as the grouping key
+                    hostname, is_dev = self._extract_node_info(result)
+                    fs_type, block_size, config_key = self._extract_filesystem_config(
+                        result
                     )
-                    html.append(
-                        f"            <tr><td>Insert Rate Range</td><td>{np.min(insert_rates):.2f} - {np.max(insert_rates):.2f} vectors/sec</td></tr>"
-                    )
-                    html.append("        </table>")
 
-                # Index performance
-                index_times = [
-                    r.get("index_performance", {}).get("creation_time_seconds", 0)
-                    for r in self.results_data
-                ]
-                if index_times and any(t > 0 for t in index_times):
-                    html.append("        <h3>🔗 Index Creation Performance</h3>")
-                    html.append("        <table class='metric-table'>")
-                    html.append(
-                        f"            <tr><td>Average Index Creation Time</td><td>{np.mean(index_times):.2f} seconds</td></tr>"
+                    if hostname not in node_performance:
+                        node_performance[hostname] = {
+                            "hostname": hostname,
+                            "node_type": "Development" if is_dev else "Baseline",
+                            "insert_rates": [],
+                            "insert_times": [],
+                            "index_times": [],
+                            "query_performance": {},
+                            "filesystem": fs_type,
+                            "block_size": block_size,
+                        }
+
+                    # Add insert performance
+                    insert_perf = result.get("insert_performance", {})
+                    if insert_perf:
+                        rate = insert_perf.get("vectors_per_second", 0)
+                        time = insert_perf.get("total_time_seconds", 0)
+                        if rate > 0:
+                            node_performance[hostname]["insert_rates"].append(rate)
+                        if time > 0:
+                            node_performance[hostname]["insert_times"].append(time)
+
+                    # Add index performance
+                    index_perf = result.get("index_performance", {})
+                    if index_perf:
+                        time = index_perf.get("creation_time_seconds", 0)
+                        if time > 0:
+                            node_performance[hostname]["index_times"].append(time)
+
+                    # Collect query performance (use first result for each node)
+                    query_perf = result.get("query_performance", {})
+                    if (
+                        query_perf
+                        and not node_performance[hostname]["query_performance"]
+                    ):
+                        node_performance[hostname]["query_performance"] = query_perf
+
+                # Display results for each node, sorted with baseline first
+                sorted_nodes = sorted(
+                    node_performance.items(),
+                    key=lambda x: (x[1]["node_type"] != "Baseline", x[0]),
+                )
+                for hostname, perf_data in sorted_nodes:
+                    node_type_badge = (
+                        "🔵" if perf_data["node_type"] == "Development" else "🟢"
                     )
                     html.append(
-                        f"            <tr><td>Index Time Range</td><td>{np.min(index_times):.2f} - {np.max(index_times):.2f} seconds</td></tr>"
+                        f"        <h3>{node_type_badge} {hostname} ({perf_data['node_type']})</h3>"
                     )
-                    html.append("        </table>")
-
-                # Query performance
-                html.append("        <h3>🔍 Query Performance</h3>")
-                first_query_perf = self.results_data[0].get("query_performance", {})
-                if first_query_perf:
-                    html.append("        <table>")
                     html.append(
-                        "            <tr><th>Query Type</th><th>Batch Size</th><th>QPS</th><th>Avg Latency (ms)</th></tr>"
+                        f"        <p>Filesystem: {perf_data['filesystem']}, Block Size: {perf_data['block_size']}</p>"
                     )
 
-                    for topk, topk_data in first_query_perf.items():
-                        for batch, batch_data in topk_data.items():
-                            qps = batch_data.get("queries_per_second", 0)
-                            avg_time = batch_data.get("average_time_seconds", 0) * 1000
-
-                            # Color coding for performance
-                            qps_class = ""
-                            if qps > 1000:
-                                qps_class = "performance-good"
-                            elif qps > 100:
-                                qps_class = "performance-warning"
-                            else:
-                                qps_class = "performance-poor"
-
-                            html.append(f"            <tr>")
-                            html.append(
-                                f"                <td>{topk.replace('topk_', 'Top-')}</td>"
-                            )
-                            html.append(
-                                f"                <td>{batch.replace('batch_', 'Batch ')}</td>"
-                            )
-                            html.append(
-                                f"                <td class='{qps_class}'>{qps:.2f}</td>"
-                            )
-                            html.append(f"                <td>{avg_time:.2f}</td>")
-                            html.append(f"            </tr>")
+                    # Insert performance
+                    insert_rates = perf_data["insert_rates"]
+                    if insert_rates:
+                        html.append("        <h4>📈 Vector Insert Performance</h4>")
+                        html.append("        <table class='metric-table'>")
+                        html.append(
+                            f"            <tr><td>Average Insert Rate</td><td>{np.mean(insert_rates):.2f} vectors/sec</td></tr>"
+                        )
+                        html.append(
+                            f"            <tr><td>Insert Rate Range</td><td>{np.min(insert_rates):.2f} - {np.max(insert_rates):.2f} vectors/sec</td></tr>"
+                        )
+                        html.append(
+                            f"            <tr><td>Test Iterations</td><td>{len(insert_rates)}</td></tr>"
+                        )
+                        html.append("        </table>")
+
+                    # Index performance
+                    index_times = perf_data["index_times"]
+                    if index_times:
+                        html.append("        <h4>🔗 Index Creation Performance</h4>")
+                        html.append("        <table class='metric-table'>")
+                        html.append(
+                            f"            <tr><td>Average Index Creation Time</td><td>{np.mean(index_times):.3f} seconds</td></tr>"
+                        )
+                        html.append(
+                            f"            <tr><td>Index Time Range</td><td>{np.min(index_times):.3f} - {np.max(index_times):.3f} seconds</td></tr>"
+                        )
+                        html.append("        </table>")
+
+                    # Query performance
+                    query_perf = perf_data["query_performance"]
+                    if query_perf:
+                        html.append("        <h4>🔍 Query Performance</h4>")
+                        html.append("        <table>")
+                        html.append(
+                            "            <tr><th>Query Type</th><th>Batch Size</th><th>QPS</th><th>Avg Latency (ms)</th></tr>"
+                        )
 
-                    html.append("        </table>")
+                        for topk, topk_data in query_perf.items():
+                            for batch, batch_data in topk_data.items():
+                                qps = batch_data.get("queries_per_second", 0)
+                                avg_time = (
+                                    batch_data.get("average_time_seconds", 0) * 1000
+                                )
+
+                                # Color coding for performance
+                                qps_class = ""
+                                if qps > 1000:
+                                    qps_class = "performance-good"
+                                elif qps > 100:
+                                    qps_class = "performance-warning"
+                                else:
+                                    qps_class = "performance-poor"
+
+                                html.append(f"            <tr>")
+                                html.append(
+                                    f"                <td>{topk.replace('topk_', 'Top-')}</td>"
+                                )
+                                html.append(
+                                    f"                <td>{batch.replace('batch_', 'Batch ')}</td>"
+                                )
+                                html.append(
+                                    f"                <td class='{qps_class}'>{qps:.2f}</td>"
+                                )
+                                html.append(f"                <td>{avg_time:.2f}</td>")
+                                html.append(f"            </tr>")
+                        html.append("        </table>")
+
+                    html.append("        <br>")  # Add spacing between configurations
 
-                html.append("    </div>")
+            html.append("    </div>")
 
             # Footer
+            # Performance Graphs Section
+            html.append("    <div class='section'>")
+            html.append("        <h2>📈 Performance Visualizations</h2>")
+            html.append(
+                "        <p>The following graphs provide visual analysis of the benchmark results across all tested filesystem configurations:</p>"
+            )
+            html.append("        <ul>")
+            html.append(
+                "            <li><strong>Insert Performance:</strong> Shows vector insertion rates and times for each filesystem configuration</li>"
+            )
+            html.append(
+                "            <li><strong>Query Performance:</strong> Displays query performance heatmaps for different Top-K and batch sizes</li>"
+            )
+            html.append(
+                "            <li><strong>Index Performance:</strong> Compares index creation times across filesystems</li>"
+            )
+            html.append(
+                "            <li><strong>Performance Matrix:</strong> Comprehensive comparison matrix of all metrics</li>"
+            )
+            html.append(
+                "            <li><strong>Filesystem Comparison:</strong> Side-by-side comparison of filesystem performance</li>"
+            )
+            html.append("        </ul>")
+            html.append(
+                "        <p><em>Note: Graphs are generated as separate PNG files in the same directory as this report.</em></p>"
+            )
+            html.append("        <div style='margin-top: 20px;'>")
+            html.append(
+                "            <img src='insert_performance.png' alt='Insert Performance' style='max-width: 100%; height: auto; margin-bottom: 20px;'>"
+            )
+            html.append(
+                "            <img src='query_performance.png' alt='Query Performance' style='max-width: 100%; height: auto; margin-bottom: 20px;'>"
+            )
+            html.append(
+                "            <img src='index_performance.png' alt='Index Performance' style='max-width: 100%; height: auto; margin-bottom: 20px;'>"
+            )
+            html.append(
+                "            <img src='performance_matrix.png' alt='Performance Matrix' style='max-width: 100%; height: auto; margin-bottom: 20px;'>"
+            )
+            html.append(
+                "            <img src='filesystem_comparison.png' alt='Filesystem Comparison' style='max-width: 100%; height: auto; margin-bottom: 20px;'>"
+            )
+            html.append("        </div>")
+            html.append("    </div>")
+
             html.append("    <div class='section'>")
             html.append("        <h2>📝 Notes</h2>")
             html.append("        <ul>")
@@ -661,10 +868,11 @@ class ResultsAnalyzer:
             return "\n".join(html)
 
         except Exception as e:
-            self.logger.error(f"Error generating HTML report: {e}")
-            return (
-                f"<html><body><h1>Error generating HTML report: {e}</h1></body></html>"
-            )
+            import traceback
+
+            tb = traceback.format_exc()
+            self.logger.error(f"Error generating HTML report: {e}\n{tb}")
+            return f"<html><body><h1>Error generating HTML report: {e}</h1><pre>{tb}</pre></body></html>"
 
     def generate_graphs(self) -> bool:
         """Generate performance visualization graphs"""
@@ -691,6 +899,9 @@ class ResultsAnalyzer:
             # Graph 4: Performance Comparison Matrix
             self._plot_performance_matrix()
 
+            # Graph 5: Multi-filesystem Comparison (if applicable)
+            self._plot_filesystem_comparison()
+
             self.logger.info("Graphs generated successfully")
             return True
 
@@ -699,34 +910,188 @@ class ResultsAnalyzer:
             return False
 
     def _plot_insert_performance(self):
-        """Plot insert performance metrics"""
-        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        """Plot insert performance metrics with node differentiation"""
+        # Group data by node
+        node_performance = {}
 
-        # Extract insert data
-        iterations = []
-        insert_rates = []
-        insert_times = []
+        for result in self.results_data:
+            hostname, is_dev = self._extract_node_info(result)
+
+            if hostname not in node_performance:
+                node_performance[hostname] = {
+                    "insert_rates": [],
+                    "insert_times": [],
+                    "iterations": [],
+                    "is_dev": is_dev,
+                }
 
-        for i, result in enumerate(self.results_data):
             insert_perf = result.get("insert_performance", {})
             if insert_perf:
-                iterations.append(i + 1)
-                insert_rates.append(insert_perf.get("vectors_per_second", 0))
-                insert_times.append(insert_perf.get("total_time_seconds", 0))
-
-        # Plot insert rate
-        ax1.plot(iterations, insert_rates, "b-o", linewidth=2, markersize=6)
-        ax1.set_xlabel("Iteration")
-        ax1.set_ylabel("Vectors/Second")
-        ax1.set_title("Vector Insert Rate Performance")
-        ax1.grid(True, alpha=0.3)
-
-        # Plot insert time
-        ax2.plot(iterations, insert_times, "r-o", linewidth=2, markersize=6)
-        ax2.set_xlabel("Iteration")
-        ax2.set_ylabel("Total Time (seconds)")
-        ax2.set_title("Vector Insert Time Performance")
-        ax2.grid(True, alpha=0.3)
+                node_performance[hostname]["insert_rates"].append(
+                    insert_perf.get("vectors_per_second", 0)
+                )
+                node_performance[hostname]["insert_times"].append(
+                    insert_perf.get("total_time_seconds", 0)
+                )
+                node_performance[hostname]["iterations"].append(
+                    len(node_performance[hostname]["insert_rates"])
+                )
+
+        # Check if we have multiple nodes
+        if len(node_performance) > 1:
+            # Multi-node mode: separate lines for each node
+            fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 7))
+
+            # Sort nodes with baseline first, then dev
+            sorted_nodes = sorted(
+                node_performance.items(), key=lambda x: (x[1]["is_dev"], x[0])
+            )
+
+            # Create color palettes for baseline and dev nodes
+            baseline_colors = [
+                "#2E7D32",
+                "#43A047",
+                "#66BB6A",
+                "#81C784",
+                "#A5D6A7",
+                "#C8E6C9",
+            ]  # Greens
+            dev_colors = [
+                "#0D47A1",
+                "#1565C0",
+                "#1976D2",
+                "#1E88E5",
+                "#2196F3",
+                "#42A5F5",
+                "#64B5F6",
+            ]  # Blues
+
+            # Additional colors if needed
+            extra_colors = [
+                "#E65100",
+                "#F57C00",
+                "#FF9800",
+                "#FFB300",
+                "#FFC107",
+                "#FFCA28",
+            ]  # Oranges
+
+            # Line styles to cycle through
+            line_styles = ["-", "--", "-.", ":"]
+            markers = ["o", "s", "^", "v", "D", "p", "*", "h"]
+
+            baseline_idx = 0
+            dev_idx = 0
+
+            # Use different colors and styles for each node
+            for idx, (hostname, perf_data) in enumerate(sorted_nodes):
+                if not perf_data["insert_rates"]:
+                    continue
+
+                # Choose color and style based on node type and index
+                if perf_data["is_dev"]:
+                    # Development nodes - blues
+                    color = dev_colors[dev_idx % len(dev_colors)]
+                    linestyle = line_styles[
+                        (dev_idx // len(dev_colors)) % len(line_styles)
+                    ]
+                    marker = markers[4 + (dev_idx % 4)]  # Use markers 4-7 for dev
+                    label = f"{hostname} (Dev)"
+                    dev_idx += 1
+                else:
+                    # Baseline nodes - greens
+                    color = baseline_colors[baseline_idx % len(baseline_colors)]
+                    linestyle = line_styles[
+                        (baseline_idx // len(baseline_colors)) % len(line_styles)
+                    ]
+                    marker = markers[
+                        baseline_idx % 4
+                    ]  # Use first 4 markers for baseline
+                    label = f"{hostname} (Baseline)"
+                    baseline_idx += 1
+
+                iterations = list(range(1, len(perf_data["insert_rates"]) + 1))
+
+                # Plot insert rate with alpha for better visibility
+                ax1.plot(
+                    iterations,
+                    perf_data["insert_rates"],
+                    color=color,
+                    linestyle=linestyle,
+                    marker=marker,
+                    linewidth=1.5,
+                    markersize=5,
+                    label=label,
+                    alpha=0.8,
+                )
+
+                # Plot insert time
+                ax2.plot(
+                    iterations,
+                    perf_data["insert_times"],
+                    color=color,
+                    linestyle=linestyle,
+                    marker=marker,
+                    linewidth=1.5,
+                    markersize=5,
+                    label=label,
+                    alpha=0.8,
+                )
+
+            ax1.set_xlabel("Iteration")
+            ax1.set_ylabel("Vectors/Second")
+            ax1.set_title("Milvus Insert Rate by Node")
+            ax1.grid(True, alpha=0.3)
+            # Position legend outside plot area for better visibility with many nodes
+            ax1.legend(bbox_to_anchor=(1.05, 1), loc="upper left", fontsize=7, ncol=1)
+
+            ax2.set_xlabel("Iteration")
+            ax2.set_ylabel("Total Time (seconds)")
+            ax2.set_title("Milvus Insert Time by Node")
+            ax2.grid(True, alpha=0.3)
+            # Position legend outside plot area for better visibility with many nodes
+            ax2.legend(bbox_to_anchor=(1.05, 1), loc="upper left", fontsize=7, ncol=1)
+
+            plt.suptitle(
+                "Insert Performance Analysis: Baseline vs Development",
+                fontsize=14,
+                y=1.02,
+            )
+        else:
+            # Single node mode: original behavior
+            fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+
+            # Extract insert data from single node
+            hostname = list(node_performance.keys())[0] if node_performance else None
+            if hostname:
+                perf_data = node_performance[hostname]
+                iterations = list(range(1, len(perf_data["insert_rates"]) + 1))
+
+                # Plot insert rate
+                ax1.plot(
+                    iterations,
+                    perf_data["insert_rates"],
+                    "b-o",
+                    linewidth=2,
+                    markersize=6,
+                )
+                ax1.set_xlabel("Iteration")
+                ax1.set_ylabel("Vectors/Second")
+                ax1.set_title(f"Vector Insert Rate Performance - {hostname}")
+                ax1.grid(True, alpha=0.3)
+
+                # Plot insert time
+                ax2.plot(
+                    iterations,
+                    perf_data["insert_times"],
+                    "r-o",
+                    linewidth=2,
+                    markersize=6,
+                )
+                ax2.set_xlabel("Iteration")
+                ax2.set_ylabel("Total Time (seconds)")
+                ax2.set_title(f"Vector Insert Time Performance - {hostname}")
+                ax2.grid(True, alpha=0.3)
 
         plt.tight_layout()
         output_file = os.path.join(
@@ -739,52 +1104,110 @@ class ResultsAnalyzer:
         plt.close()
 
     def _plot_query_performance(self):
-        """Plot query performance metrics"""
+        """Plot query performance metrics comparing baseline vs dev nodes"""
         if not self.results_data:
             return
 
-        # Collect query performance data
-        query_data = []
+        # Group data by filesystem configuration
+        fs_groups = {}
         for result in self.results_data:
+            hostname, is_dev = self._extract_node_info(result)
+            fs_type, block_size, config_key = self._extract_filesystem_config(result)
+
+            if config_key not in fs_groups:
+                fs_groups[config_key] = {"baseline": [], "dev": []}
+
             query_perf = result.get("query_performance", {})
-            for topk, topk_data in query_perf.items():
-                for batch, batch_data in topk_data.items():
-                    query_data.append(
-                        {
-                            "topk": topk.replace("topk_", ""),
-                            "batch": batch.replace("batch_", ""),
-                            "qps": batch_data.get("queries_per_second", 0),
-                            "avg_time": batch_data.get("average_time_seconds", 0)
-                            * 1000,  # Convert to ms
-                        }
-                    )
+            if query_perf:
+                node_type = "dev" if is_dev else "baseline"
+                for topk, topk_data in query_perf.items():
+                    for batch, batch_data in topk_data.items():
+                        fs_groups[config_key][node_type].append(
+                            {
+                                "hostname": hostname,
+                                "topk": topk.replace("topk_", ""),
+                                "batch": batch.replace("batch_", ""),
+                                "qps": batch_data.get("queries_per_second", 0),
+                                "avg_time": batch_data.get("average_time_seconds", 0)
+                                * 1000,
+                            }
+                        )
 
-        if not query_data:
+        if not fs_groups:
             return
 
-        df = pd.DataFrame(query_data)
+        # Create subplots for each filesystem config
+        n_configs = len(fs_groups)
+        fig_height = max(8, 4 * n_configs)
+        fig, axes = plt.subplots(n_configs, 2, figsize=(16, fig_height))
 
-        # Create subplots
-        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        if n_configs == 1:
+            axes = axes.reshape(1, -1)
 
-        # QPS heatmap
-        qps_pivot = df.pivot_table(
-            values="qps", index="topk", columns="batch", aggfunc="mean"
-        )
-        sns.heatmap(qps_pivot, annot=True, fmt=".1f", ax=ax1, cmap="YlOrRd")
-        ax1.set_title("Queries Per Second (QPS)")
-        ax1.set_xlabel("Batch Size")
-        ax1.set_ylabel("Top-K")
-
-        # Latency heatmap
-        latency_pivot = df.pivot_table(
-            values="avg_time", index="topk", columns="batch", aggfunc="mean"
-        )
-        sns.heatmap(latency_pivot, annot=True, fmt=".1f", ax=ax2, cmap="YlOrRd")
-        ax2.set_title("Average Query Latency (ms)")
-        ax2.set_xlabel("Batch Size")
-        ax2.set_ylabel("Top-K")
+        for idx, (config_key, data) in enumerate(sorted(fs_groups.items())):
+            # Create DataFrames for baseline and dev
+            baseline_df = (
+                pd.DataFrame(data["baseline"]) if data["baseline"] else pd.DataFrame()
+            )
+            dev_df = pd.DataFrame(data["dev"]) if data["dev"] else pd.DataFrame()
+
+            # Baseline QPS heatmap
+            ax_base = axes[idx][0]
+            if not baseline_df.empty:
+                baseline_pivot = baseline_df.pivot_table(
+                    values="qps", index="topk", columns="batch", aggfunc="mean"
+                )
+                sns.heatmap(
+                    baseline_pivot,
+                    annot=True,
+                    fmt=".1f",
+                    ax=ax_base,
+                    cmap="Greens",
+                    cbar_kws={"label": "QPS"},
+                )
+                ax_base.set_title(f"{config_key.upper()} - Baseline QPS")
+                ax_base.set_xlabel("Batch Size")
+                ax_base.set_ylabel("Top-K")
+            else:
+                ax_base.text(
+                    0.5,
+                    0.5,
+                    f"No baseline data for {config_key}",
+                    ha="center",
+                    va="center",
+                    transform=ax_base.transAxes,
+                )
+                ax_base.set_title(f"{config_key.upper()} - Baseline QPS")
 
+            # Dev QPS heatmap
+            ax_dev = axes[idx][1]
+            if not dev_df.empty:
+                dev_pivot = dev_df.pivot_table(
+                    values="qps", index="topk", columns="batch", aggfunc="mean"
+                )
+                sns.heatmap(
+                    dev_pivot,
+                    annot=True,
+                    fmt=".1f",
+                    ax=ax_dev,
+                    cmap="Blues",
+                    cbar_kws={"label": "QPS"},
+                )
+                ax_dev.set_title(f"{config_key.upper()} - Development QPS")
+                ax_dev.set_xlabel("Batch Size")
+                ax_dev.set_ylabel("Top-K")
+            else:
+                ax_dev.text(
+                    0.5,
+                    0.5,
+                    f"No dev data for {config_key}",
+                    ha="center",
+                    va="center",
+                    transform=ax_dev.transAxes,
+                )
+                ax_dev.set_title(f"{config_key.upper()} - Development QPS")
+
+        plt.suptitle("Query Performance: Baseline vs Development", fontsize=16, y=1.02)
         plt.tight_layout()
         output_file = os.path.join(
             self.output_dir,
@@ -796,32 +1219,101 @@ class ResultsAnalyzer:
         plt.close()
 
     def _plot_index_performance(self):
-        """Plot index creation performance"""
-        iterations = []
-        index_times = []
+        """Plot index creation performance comparing baseline vs dev"""
+        # Group by filesystem configuration
+        fs_groups = {}
+
+        for result in self.results_data:
+            hostname, is_dev = self._extract_node_info(result)
+            fs_type, block_size, config_key = self._extract_filesystem_config(result)
+
+            if config_key not in fs_groups:
+                fs_groups[config_key] = {"baseline": [], "dev": []}
 
-        for i, result in enumerate(self.results_data):
             index_perf = result.get("index_performance", {})
             if index_perf:
-                iterations.append(i + 1)
-                index_times.append(index_perf.get("creation_time_seconds", 0))
+                time = index_perf.get("creation_time_seconds", 0)
+                if time > 0:
+                    node_type = "dev" if is_dev else "baseline"
+                    fs_groups[config_key][node_type].append(time)
 
-        if not index_times:
+        if not fs_groups:
             return
 
-        plt.figure(figsize=(10, 6))
-        plt.bar(iterations, index_times, alpha=0.7, color="green")
-        plt.xlabel("Iteration")
-        plt.ylabel("Index Creation Time (seconds)")
-        plt.title("Index Creation Performance")
-        plt.grid(True, alpha=0.3)
-
-        # Add average line
-        avg_time = np.mean(index_times)
-        plt.axhline(
-            y=avg_time, color="red", linestyle="--", label=f"Average: {avg_time:.2f}s"
+        # Create comparison bar chart
+        fig, ax = plt.subplots(figsize=(14, 8))
+
+        configs = sorted(fs_groups.keys())
+        x = np.arange(len(configs))
+        width = 0.35
+
+        # Calculate averages for each config
+        baseline_avgs = []
+        dev_avgs = []
+        baseline_stds = []
+        dev_stds = []
+
+        for config in configs:
+            baseline_times = fs_groups[config]["baseline"]
+            dev_times = fs_groups[config]["dev"]
+
+            baseline_avgs.append(np.mean(baseline_times) if baseline_times else 0)
+            dev_avgs.append(np.mean(dev_times) if dev_times else 0)
+            baseline_stds.append(np.std(baseline_times) if baseline_times else 0)
+            dev_stds.append(np.std(dev_times) if dev_times else 0)
+
+        # Create bars
+        bars1 = ax.bar(
+            x - width / 2,
+            baseline_avgs,
+            width,
+            yerr=baseline_stds,
+            label="Baseline",
+            color="#4CAF50",
+            capsize=5,
+        )
+        bars2 = ax.bar(
+            x + width / 2,
+            dev_avgs,
+            width,
+            yerr=dev_stds,
+            label="Development",
+            color="#2196F3",
+            capsize=5,
         )
-        plt.legend()
+
+        # Add value labels on bars
+        for bar, val in zip(bars1, baseline_avgs):
+            if val > 0:
+                height = bar.get_height()
+                ax.text(
+                    bar.get_x() + bar.get_width() / 2.0,
+                    height,
+                    f"{val:.3f}s",
+                    ha="center",
+                    va="bottom",
+                    fontsize=9,
+                )
+
+        for bar, val in zip(bars2, dev_avgs):
+            if val > 0:
+                height = bar.get_height()
+                ax.text(
+                    bar.get_x() + bar.get_width() / 2.0,
+                    height,
+                    f"{val:.3f}s",
+                    ha="center",
+                    va="bottom",
+                    fontsize=9,
+                )
+
+        ax.set_xlabel("Filesystem Configuration", fontsize=12)
+        ax.set_ylabel("Index Creation Time (seconds)", fontsize=12)
+        ax.set_title("Index Creation Performance: Baseline vs Development", fontsize=14)
+        ax.set_xticks(x)
+        ax.set_xticklabels([c.upper() for c in configs], rotation=45, ha="right")
+        ax.legend(loc="upper right")
+        ax.grid(True, alpha=0.3, axis="y")
 
         output_file = os.path.join(
             self.output_dir,
@@ -833,61 +1325,148 @@ class ResultsAnalyzer:
         plt.close()
 
     def _plot_performance_matrix(self):
-        """Plot comprehensive performance comparison matrix"""
+        """Plot performance comparison matrix for each filesystem config"""
         if len(self.results_data) < 2:
             return
 
-        # Extract key metrics for comparison
-        metrics = []
-        for i, result in enumerate(self.results_data):
+        # Group by filesystem configuration
+        fs_metrics = {}
+
+        for result in self.results_data:
+            hostname, is_dev = self._extract_node_info(result)
+            fs_type, block_size, config_key = self._extract_filesystem_config(result)
+
+            if config_key not in fs_metrics:
+                fs_metrics[config_key] = {"baseline": [], "dev": []}
+
+            # Collect metrics
             insert_perf = result.get("insert_performance", {})
             index_perf = result.get("index_performance", {})
+            query_perf = result.get("query_performance", {})
 
             metric = {
-                "iteration": i + 1,
+                "hostname": hostname,
                 "insert_rate": insert_perf.get("vectors_per_second", 0),
                 "index_time": index_perf.get("creation_time_seconds", 0),
             }
 
-            # Add query metrics
-            query_perf = result.get("query_performance", {})
+            # Get representative query performance (topk_10, batch_1)
             if "topk_10" in query_perf and "batch_1" in query_perf["topk_10"]:
                 metric["query_qps"] = query_perf["topk_10"]["batch_1"].get(
                     "queries_per_second", 0
                 )
+            else:
+                metric["query_qps"] = 0
 
-            metrics.append(metric)
+            node_type = "dev" if is_dev else "baseline"
+            fs_metrics[config_key][node_type].append(metric)
 
-        df = pd.DataFrame(metrics)
+        if not fs_metrics:
+            return
 
-        # Normalize metrics for comparison
-        numeric_cols = ["insert_rate", "index_time", "query_qps"]
-        for col in numeric_cols:
-            if col in df.columns:
-                df[f"{col}_norm"] = (df[col] - df[col].min()) / (
-                    df[col].max() - df[col].min() + 1e-6
-                )
+        # Create subplots for each filesystem
+        n_configs = len(fs_metrics)
+        n_cols = min(3, n_configs)
+        n_rows = (n_configs + n_cols - 1) // n_cols
+
+        fig, axes = plt.subplots(n_rows, n_cols, figsize=(n_cols * 6, n_rows * 5))
+        if n_rows == 1 and n_cols == 1:
+            axes = [[axes]]
+        elif n_rows == 1:
+            axes = [axes]
+        elif n_cols == 1:
+            axes = [[ax] for ax in axes]
+
+        for idx, (config_key, data) in enumerate(sorted(fs_metrics.items())):
+            row = idx // n_cols
+            col = idx % n_cols
+            ax = axes[row][col]
+
+            # Calculate averages
+            baseline_metrics = data["baseline"]
+            dev_metrics = data["dev"]
+
+            if baseline_metrics and dev_metrics:
+                categories = ["Insert Rate\n(vec/s)", "Index Time\n(s)", "Query QPS"]
+
+                baseline_avg = [
+                    np.mean([m["insert_rate"] for m in baseline_metrics]),
+                    np.mean([m["index_time"] for m in baseline_metrics]),
+                    np.mean([m["query_qps"] for m in baseline_metrics]),
+                ]
 
-        # Create radar chart
-        fig, ax = plt.subplots(figsize=(10, 8), subplot_kw=dict(projection="polar"))
+                dev_avg = [
+                    np.mean([m["insert_rate"] for m in dev_metrics]),
+                    np.mean([m["index_time"] for m in dev_metrics]),
+                    np.mean([m["query_qps"] for m in dev_metrics]),
+                ]
 
-        angles = np.linspace(0, 2 * np.pi, len(numeric_cols), endpoint=False).tolist()
-        angles += angles[:1]  # Complete the circle
+                x = np.arange(len(categories))
+                width = 0.35
 
-        for i, row in df.iterrows():
-            values = [row.get(f"{col}_norm", 0) for col in numeric_cols]
-            values += values[:1]  # Complete the circle
+                bars1 = ax.bar(
+                    x - width / 2,
+                    baseline_avg,
+                    width,
+                    label="Baseline",
+                    color="#4CAF50",
+                )
+                bars2 = ax.bar(
+                    x + width / 2, dev_avg, width, label="Development", color="#2196F3"
+                )
 
-            ax.plot(
-                angles, values, "o-", linewidth=2, label=f'Iteration {row["iteration"]}'
-            )
-            ax.fill(angles, values, alpha=0.25)
+                # Add value labels
+                for bar, val in zip(bars1, baseline_avg):
+                    height = bar.get_height()
+                    ax.text(
+                        bar.get_x() + bar.get_width() / 2.0,
+                        height,
+                        f"{val:.0f}" if val > 100 else f"{val:.2f}",
+                        ha="center",
+                        va="bottom",
+                        fontsize=8,
+                    )
 
-        ax.set_xticks(angles[:-1])
-        ax.set_xticklabels(["Insert Rate", "Index Time (inv)", "Query QPS"])
-        ax.set_ylim(0, 1)
-        ax.set_title("Performance Comparison Matrix (Normalized)", y=1.08)
-        ax.legend(loc="upper right", bbox_to_anchor=(1.3, 1.0))
+                for bar, val in zip(bars2, dev_avg):
+                    height = bar.get_height()
+                    ax.text(
+                        bar.get_x() + bar.get_width() / 2.0,
+                        height,
+                        f"{val:.0f}" if val > 100 else f"{val:.2f}",
+                        ha="center",
+                        va="bottom",
+                        fontsize=8,
+                    )
+
+                ax.set_xlabel("Metrics")
+                ax.set_ylabel("Value")
+                ax.set_title(f"{config_key.upper()}")
+                ax.set_xticks(x)
+                ax.set_xticklabels(categories)
+                ax.legend(loc="upper right", fontsize=8)
+                ax.grid(True, alpha=0.3, axis="y")
+            else:
+                ax.text(
+                    0.5,
+                    0.5,
+                    f"Insufficient data\nfor {config_key}",
+                    ha="center",
+                    va="center",
+                    transform=ax.transAxes,
+                )
+                ax.set_title(f"{config_key.upper()}")
+
+        # Hide unused subplots
+        for idx in range(n_configs, n_rows * n_cols):
+            row = idx // n_cols
+            col = idx % n_cols
+            axes[row][col].set_visible(False)
+
+        plt.suptitle(
+            "Performance Comparison Matrix: Baseline vs Development",
+            fontsize=14,
+            y=1.02,
+        )
 
         output_file = os.path.join(
             self.output_dir,
@@ -898,6 +1477,149 @@ class ResultsAnalyzer:
         )
         plt.close()
 
+    def _plot_filesystem_comparison(self):
+        """Plot node performance comparison chart"""
+        if len(self.results_data) < 2:
+            return
+
+        # Group results by node
+        node_performance = {}
+
+        for result in self.results_data:
+            hostname, is_dev = self._extract_node_info(result)
+
+            if hostname not in node_performance:
+                node_performance[hostname] = {
+                    "insert_rates": [],
+                    "index_times": [],
+                    "query_qps": [],
+                    "is_dev": is_dev,
+                }
+
+            # Collect metrics
+            insert_perf = result.get("insert_performance", {})
+            if insert_perf:
+                node_performance[hostname]["insert_rates"].append(
+                    insert_perf.get("vectors_per_second", 0)
+                )
+
+            index_perf = result.get("index_performance", {})
+            if index_perf:
+                node_performance[hostname]["index_times"].append(
+                    index_perf.get("creation_time_seconds", 0)
+                )
+
+            # Get top-10 batch-1 query performance as representative
+            query_perf = result.get("query_performance", {})
+            if "topk_10" in query_perf and "batch_1" in query_perf["topk_10"]:
+                qps = query_perf["topk_10"]["batch_1"].get("queries_per_second", 0)
+                node_performance[hostname]["query_qps"].append(qps)
+
+        # Only create comparison if we have multiple nodes
+        if len(node_performance) > 1:
+            # Calculate averages
+            node_metrics = {}
+            for hostname, perf_data in node_performance.items():
+                node_metrics[hostname] = {
+                    "avg_insert_rate": (
+                        np.mean(perf_data["insert_rates"])
+                        if perf_data["insert_rates"]
+                        else 0
+                    ),
+                    "avg_index_time": (
+                        np.mean(perf_data["index_times"])
+                        if perf_data["index_times"]
+                        else 0
+                    ),
+                    "avg_query_qps": (
+                        np.mean(perf_data["query_qps"]) if perf_data["query_qps"] else 0
+                    ),
+                    "is_dev": perf_data["is_dev"],
+                }
+
+            # Create comparison bar chart with more space
+            fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(24, 8))
+
+            # Sort nodes with baseline first
+            sorted_nodes = sorted(
+                node_metrics.items(), key=lambda x: (x[1]["is_dev"], x[0])
+            )
+            node_names = [hostname for hostname, _ in sorted_nodes]
+
+            # Use different colors for baseline vs dev
+            colors = [
+                "#4CAF50" if not node_metrics[hostname]["is_dev"] else "#2196F3"
+                for hostname in node_names
+            ]
+
+            # Add labels for clarity
+            labels = [
+                f"{hostname}\n({'Dev' if node_metrics[hostname]['is_dev'] else 'Baseline'})"
+                for hostname in node_names
+            ]
+
+            # Insert rate comparison
+            insert_rates = [
+                node_metrics[hostname]["avg_insert_rate"] for hostname in node_names
+            ]
+            bars1 = ax1.bar(labels, insert_rates, color=colors)
+            ax1.set_title("Average Milvus Insert Rate by Node")
+            ax1.set_ylabel("Vectors/Second")
+            # Rotate labels for better readability
+            ax1.set_xticklabels(labels, rotation=45, ha="right", fontsize=8)
+
+            # Index time comparison (lower is better)
+            index_times = [
+                node_metrics[hostname]["avg_index_time"] for hostname in node_names
+            ]
+            bars2 = ax2.bar(labels, index_times, color=colors)
+            ax2.set_title("Average Milvus Index Time by Node")
+            ax2.set_ylabel("Seconds (Lower is Better)")
+            ax2.set_xticklabels(labels, rotation=45, ha="right", fontsize=8)
+
+            # Query QPS comparison
+            query_qps = [
+                node_metrics[hostname]["avg_query_qps"] for hostname in node_names
+            ]
+            bars3 = ax3.bar(labels, query_qps, color=colors)
+            ax3.set_title("Average Milvus Query QPS by Node")
+            ax3.set_ylabel("Queries/Second")
+            ax3.set_xticklabels(labels, rotation=45, ha="right", fontsize=8)
+
+            # Add value labels on bars
+            for bars, values in [
+                (bars1, insert_rates),
+                (bars2, index_times),
+                (bars3, query_qps),
+            ]:
+                for bar, value in zip(bars, values):
+                    height = bar.get_height()
+                    ax = bar.axes
+                    ax.text(
+                        bar.get_x() + bar.get_width() / 2.0,
+                        height + height * 0.01,
+                        f"{value:.1f}",
+                        ha="center",
+                        va="bottom",
+                        fontsize=10,
+                    )
+
+            plt.suptitle(
+                "Milvus Performance Comparison: Baseline vs Development Nodes",
+                fontsize=16,
+                y=1.02,
+            )
+            plt.tight_layout()
+
+            output_file = os.path.join(
+                self.output_dir,
+                f"filesystem_comparison.{self.config.get('graph_format', 'png')}",
+            )
+            plt.savefig(
+                output_file, dpi=self.config.get("graph_dpi", 300), bbox_inches="tight"
+            )
+            plt.close()
+
     def analyze(self) -> bool:
         """Run complete analysis"""
         self.logger.info("Starting results analysis...")
playbooks/roles/ai_collect_results/files/generate_better_graphs.py +9 -7
index 645bac9e..b3681ff9 100755
@@ -29,17 +29,18 @@ def extract_filesystem_from_filename(filename):
         if "_" in node_name:
             parts = node_name.split("_")
             node_name = "_".join(parts[:-1])  # Remove last part (iteration)
-        
+
         # Extract filesystem type from node name
         if "-xfs-" in node_name:
             return "xfs"
         elif "-ext4-" in node_name:
-            return "ext4"  
+            return "ext4"
         elif "-btrfs-" in node_name:
             return "btrfs"
-    
+
     return "unknown"
 
+
 def extract_node_config_from_filename(filename):
     """Extract detailed node configuration from filename"""
     # Expected format: results_debian13-ai-xfs-4k-4ks_1.json
@@ -50,14 +51,15 @@ def extract_node_config_from_filename(filename):
         if "_" in node_name:
             parts = node_name.split("_")
             node_name = "_".join(parts[:-1])  # Remove last part (iteration)
-        
+
         # Remove -dev suffix if present
         node_name = node_name.replace("-dev", "")
-        
+
         return node_name.replace("debian13-ai-", "")
-    
+
     return "unknown"
 
+
 def detect_filesystem():
     """Detect the filesystem type of /data on test nodes"""
     # This is now a fallback - we primarily use filename-based detection
@@ -104,7 +106,7 @@ def load_results(results_dir):
                 # Extract node type from filename
                 filename = os.path.basename(json_file)
                 data["filename"] = filename
-                
+
                 # Extract filesystem type and config from filename
                 data["filesystem"] = extract_filesystem_from_filename(filename)
                 data["node_config"] = extract_node_config_from_filename(filename)
playbooks/roles/ai_collect_results/files/generate_graphs.py +286 -602
index 53a835e2..fafc62bf 100755
@@ -9,7 +9,6 @@ import sys
 import glob
 import numpy as np
 import matplotlib
-
 matplotlib.use("Agg")  # Use non-interactive backend
 import matplotlib.pyplot as plt
 from datetime import datetime
@@ -17,68 +16,78 @@ from pathlib import Path
 from collections import defaultdict
 
 
+def _extract_filesystem_config(result):
+    """Extract filesystem type and block size from result data.
+    Returns (fs_type, block_size, config_key)"""
+    filename = result.get("_file", "")
+
+    # Primary: Extract filesystem type from filename (more reliable than JSON)
+    fs_type = "unknown"
+    block_size = "default"
+
+    if "xfs" in filename:
+        fs_type = "xfs"
+        # Check larger sizes first to avoid substring matches
+        if "64k" in filename and "64k-" in filename:
+            block_size = "64k"
+        elif "32k" in filename and "32k-" in filename:
+            block_size = "32k"
+        elif "16k" in filename and "16k-" in filename:
+            block_size = "16k"
+        elif "4k" in filename and "4k-" in filename:
+            block_size = "4k"
+    elif "ext4" in filename:
+        fs_type = "ext4"
+        if "4k" in filename and "4k-" in filename:
+            block_size = "4k"
+        elif "16k" in filename and "16k-" in filename:
+            block_size = "16k"
+    elif "btrfs" in filename:
+        fs_type = "btrfs"
+
+    # Fallback: Check JSON data if filename parsing failed
+    if fs_type == "unknown":
+        fs_type = result.get("filesystem", "unknown")
+
+    # Create descriptive config key
+    config_key = f"{fs_type}-{block_size}" if block_size != "default" else fs_type
+    return fs_type, block_size, config_key
+
+
+def _extract_node_info(result):
+    """Extract node hostname and determine if it's a dev node.
+    Returns (hostname, is_dev_node)"""
+    # Get hostname from system_info (preferred) or fall back to filename
+    system_info = result.get("system_info", {})
+    hostname = system_info.get("hostname", "")
+    
+    # If no hostname in system_info, try extracting from filename
+    if not hostname:
+        filename = result.get("_file", "")
+        # Remove results_ prefix and .json suffix
+        hostname = filename.replace("results_", "").replace(".json", "")
+        # Remove iteration number if present (_1, _2, etc.)
+        if "_" in hostname and hostname.split("_")[-1].isdigit():
+            hostname = "_".join(hostname.split("_")[:-1])
+    
+    # Determine if this is a dev node
+    is_dev = hostname.endswith("-dev")
+    
+    return hostname, is_dev
+
+
 def load_results(results_dir):
     """Load all JSON result files from the directory"""
     results = []
-    json_files = glob.glob(os.path.join(results_dir, "*.json"))
+    # Only load results_*.json files, not consolidated or other JSON files
+    json_files = glob.glob(os.path.join(results_dir, "results_*.json"))
 
     for json_file in json_files:
         try:
             with open(json_file, "r") as f:
                 data = json.load(f)
-                # Extract filesystem info - prefer from JSON data over filename
-                filename = os.path.basename(json_file)
-                
-                # First, try to get filesystem from the JSON data itself
-                fs_type = data.get("filesystem", None)
-                
-                # If not in JSON, try to parse from filename (backwards compatibility)
-                if not fs_type:
-                    parts = filename.replace("results_", "").replace(".json", "").split("-")
-                    
-                    # Parse host info
-                    if "debian13-ai-" in filename:
-                        host_parts = (
-                            filename.replace("results_debian13-ai-", "")
-                            .replace("_1.json", "")
-                            .replace("_2.json", "")
-                            .replace("_3.json", "")
-                            .split("-")
-                        )
-                        if "xfs" in host_parts[0]:
-                            fs_type = "xfs"
-                            # Extract block size (e.g., "4k", "16k", etc.)
-                            block_size = host_parts[1] if len(host_parts) > 1 else "unknown"
-                        elif "ext4" in host_parts[0]:
-                            fs_type = "ext4"
-                            block_size = host_parts[1] if len(host_parts) > 1 else "4k"
-                        elif "btrfs" in host_parts[0]:
-                            fs_type = "btrfs"
-                            block_size = "default"
-                        else:
-                            fs_type = "unknown"
-                            block_size = "unknown"
-                    else:
-                        fs_type = "unknown"
-                        block_size = "unknown"
-                else:
-                    # If filesystem came from JSON, set appropriate block size
-                    if fs_type == "btrfs":
-                        block_size = "default"
-                    elif fs_type in ["ext4", "xfs"]:
-                        block_size = data.get("block_size", "4k")
-                    else:
-                        block_size = data.get("block_size", "default")
-                
-                is_dev = "dev" in filename
-                
-                # Use filesystem from JSON if available, otherwise use parsed value
-                if "filesystem" not in data:
-                    data["filesystem"] = fs_type
-                data["block_size"] = block_size
-                data["is_dev"] = is_dev
-                data["filename"] = filename
-
+                # Add filename for filesystem detection
+                data["_file"] = os.path.basename(json_file)
                 results.append(data)
         except Exception as e:
             print(f"Error loading {json_file}: {e}")
@@ -86,554 +95,243 @@ def load_results(results_dir):
     return results
 
 
-def create_filesystem_comparison_chart(results, output_dir):
-    """Create a bar chart comparing performance across filesystems"""
-    # Group by filesystem and baseline/dev
-    fs_data = defaultdict(lambda: {"baseline": [], "dev": []})
-
-    for result in results:
-        fs = result.get("filesystem", "unknown")
-        category = "dev" if result.get("is_dev", False) else "baseline"
-
-        # Extract actual performance data from results
-        if "insert_performance" in result:
-            insert_qps = result["insert_performance"].get("vectors_per_second", 0)
-        else:
-            insert_qps = 0
-        fs_data[fs][category].append(insert_qps)
-
-    # Prepare data for plotting
-    filesystems = list(fs_data.keys())
-    baseline_means = [
-        np.mean(fs_data[fs]["baseline"]) if fs_data[fs]["baseline"] else 0
-        for fs in filesystems
-    ]
-    dev_means = [
-        np.mean(fs_data[fs]["dev"]) if fs_data[fs]["dev"] else 0 for fs in filesystems
-    ]
-
-    x = np.arange(len(filesystems))
-    width = 0.35
-
-    fig, ax = plt.subplots(figsize=(10, 6))
-    baseline_bars = ax.bar(
-        x - width / 2, baseline_means, width, label="Baseline", color="#1f77b4"
-    )
-    dev_bars = ax.bar(
-        x + width / 2, dev_means, width, label="Development", color="#ff7f0e"
-    )
-
-    ax.set_xlabel("Filesystem")
-    ax.set_ylabel("Insert QPS")
-    ax.set_title("Vector Database Performance by Filesystem")
-    ax.set_xticks(x)
-    ax.set_xticklabels(filesystems)
-    ax.legend()
-    ax.grid(True, alpha=0.3)
-
-    # Add value labels on bars
-    for bars in [baseline_bars, dev_bars]:
-        for bar in bars:
-            height = bar.get_height()
-            if height > 0:
-                ax.annotate(
-                    f"{height:.0f}",
-                    xy=(bar.get_x() + bar.get_width() / 2, height),
-                    xytext=(0, 3),
-                    textcoords="offset points",
-                    ha="center",
-                    va="bottom",
-                )
-
-    plt.tight_layout()
-    plt.savefig(os.path.join(output_dir, "filesystem_comparison.png"), dpi=150)
-    plt.close()
-
-
-def create_block_size_analysis(results, output_dir):
-    """Create analysis for different block sizes (XFS specific)"""
-    # Filter XFS results
-    xfs_results = [r for r in results if r.get("filesystem") == "xfs"]
-
-    if not xfs_results:
+def create_simple_performance_trends(results, output_dir):
+    """Create multi-node performance trends chart"""
+    if not results:
         return
 
-    # Group by block size
-    block_size_data = defaultdict(lambda: {"baseline": [], "dev": []})
-
-    for result in xfs_results:
-        block_size = result.get("block_size", "unknown")
-        category = "dev" if result.get("is_dev", False) else "baseline"
-        if "insert_performance" in result:
-            insert_qps = result["insert_performance"].get("vectors_per_second", 0)
-        else:
-            insert_qps = 0
-        block_size_data[block_size][category].append(insert_qps)
-
-    # Sort block sizes
-    block_sizes = sorted(
-        block_size_data.keys(),
-        key=lambda x: (
-            int(x.replace("k", "").replace("s", ""))
-            if x not in ["unknown", "default"]
-            else 0
-        ),
-    )
-
-    # Create grouped bar chart
-    baseline_means = [
-        (
-            np.mean(block_size_data[bs]["baseline"])
-            if block_size_data[bs]["baseline"]
-            else 0
-        )
-        for bs in block_sizes
-    ]
-    dev_means = [
-        np.mean(block_size_data[bs]["dev"]) if block_size_data[bs]["dev"] else 0
-        for bs in block_sizes
-    ]
-
-    x = np.arange(len(block_sizes))
-    width = 0.35
-
-    fig, ax = plt.subplots(figsize=(12, 6))
-    baseline_bars = ax.bar(
-        x - width / 2, baseline_means, width, label="Baseline", color="#2ca02c"
-    )
-    dev_bars = ax.bar(
-        x + width / 2, dev_means, width, label="Development", color="#d62728"
-    )
-
-    ax.set_xlabel("Block Size")
-    ax.set_ylabel("Insert QPS")
-    ax.set_title("XFS Performance by Block Size")
-    ax.set_xticks(x)
-    ax.set_xticklabels(block_sizes)
-    ax.legend()
-    ax.grid(True, alpha=0.3)
-
-    # Add value labels
-    for bars in [baseline_bars, dev_bars]:
-        for bar in bars:
-            height = bar.get_height()
-            if height > 0:
-                ax.annotate(
-                    f"{height:.0f}",
-                    xy=(bar.get_x() + bar.get_width() / 2, height),
-                    xytext=(0, 3),
-                    textcoords="offset points",
-                    ha="center",
-                    va="bottom",
-                )
-
-    plt.tight_layout()
-    plt.savefig(os.path.join(output_dir, "xfs_block_size_analysis.png"), dpi=150)
-    plt.close()
-
-
-def create_heatmap_analysis(results, output_dir):
-    """Create a heatmap showing performance across all configurations"""
-    # Group data by configuration and version
-    config_data = defaultdict(
-        lambda: {
-            "baseline": {"insert": 0, "query": 0},
-            "dev": {"insert": 0, "query": 0},
-        }
-    )
+    # Group results by node
+    node_performance = defaultdict(lambda: {
+        "insert_rates": [],
+        "insert_times": [],
+        "iterations": [],
+        "is_dev": False,
+    })
 
     for result in results:
-        fs = result.get("filesystem", "unknown")
-        block_size = result.get("block_size", "default")
-        config = f"{fs}-{block_size}"
-        version = "dev" if result.get("is_dev", False) else "baseline"
-
-        # Get actual insert performance
-        if "insert_performance" in result:
-            insert_qps = result["insert_performance"].get("vectors_per_second", 0)
-        else:
-            insert_qps = 0
-
-        # Calculate average query QPS
-        query_qps = 0
-        if "query_performance" in result:
-            qp = result["query_performance"]
-            total_qps = 0
-            count = 0
-            for topk_key in ["topk_1", "topk_10", "topk_100"]:
-                if topk_key in qp:
-                    for batch_key in ["batch_1", "batch_10", "batch_100"]:
-                        if batch_key in qp[topk_key]:
-                            total_qps += qp[topk_key][batch_key].get(
-                                "queries_per_second", 0
-                            )
-                            count += 1
-            if count > 0:
-                query_qps = total_qps / count
-
-        config_data[config][version]["insert"] = insert_qps
-        config_data[config][version]["query"] = query_qps
-
-    # Sort configurations
-    configs = sorted(config_data.keys())
-
-    # Prepare data for heatmap
-    insert_baseline = [config_data[c]["baseline"]["insert"] for c in configs]
-    insert_dev = [config_data[c]["dev"]["insert"] for c in configs]
-    query_baseline = [config_data[c]["baseline"]["query"] for c in configs]
-    query_dev = [config_data[c]["dev"]["query"] for c in configs]
-
-    # Create figure with custom heatmap
-    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 8))
-
-    # Create data matrices
-    insert_data = np.array([insert_baseline, insert_dev]).T
-    query_data = np.array([query_baseline, query_dev]).T
-
-    # Insert QPS heatmap
-    im1 = ax1.imshow(insert_data, cmap="YlOrRd", aspect="auto")
-    ax1.set_xticks([0, 1])
-    ax1.set_xticklabels(["Baseline", "Development"])
-    ax1.set_yticks(range(len(configs)))
-    ax1.set_yticklabels(configs)
-    ax1.set_title("Insert Performance Heatmap")
-    ax1.set_ylabel("Configuration")
-
-    # Add text annotations
-    for i in range(len(configs)):
-        for j in range(2):
-            text = ax1.text(
-                j,
-                i,
-                f"{int(insert_data[i, j])}",
-                ha="center",
-                va="center",
-                color="black",
-            )
+        hostname, is_dev = _extract_node_info(result)
+        
+        if hostname not in node_performance:
+            node_performance[hostname] = {
+                "insert_rates": [],
+                "insert_times": [],
+                "iterations": [],
+                "is_dev": is_dev,
+            }
 
-    # Add colorbar
-    cbar1 = plt.colorbar(im1, ax=ax1)
-    cbar1.set_label("Insert QPS")
-
-    # Query QPS heatmap
-    im2 = ax2.imshow(query_data, cmap="YlGnBu", aspect="auto")
-    ax2.set_xticks([0, 1])
-    ax2.set_xticklabels(["Baseline", "Development"])
-    ax2.set_yticks(range(len(configs)))
-    ax2.set_yticklabels(configs)
-    ax2.set_title("Query Performance Heatmap")
-
-    # Add text annotations
-    for i in range(len(configs)):
-        for j in range(2):
-            text = ax2.text(
-                j,
-                i,
-                f"{int(query_data[i, j])}",
-                ha="center",
-                va="center",
-                color="black",
+        insert_perf = result.get("insert_performance", {})
+        if insert_perf:
+            node_performance[hostname]["insert_rates"].append(
+                insert_perf.get("vectors_per_second", 0)
+            )
+            fs_performance[config_key]["insert_times"].append(
+                insert_perf.get("total_time_seconds", 0)
+            )
+            fs_performance[config_key]["iterations"].append(
+                len(fs_performance[config_key]["insert_rates"])
             )
 
-    # Add colorbar
-    cbar2 = plt.colorbar(im2, ax=ax2)
-    cbar2.set_label("Query QPS")
-
-    plt.tight_layout()
-    plt.savefig(os.path.join(output_dir, "performance_heatmap.png"), dpi=150)
-    plt.close()
-
-
-def create_performance_trends(results, output_dir):
-    """Create line charts showing performance trends"""
-    # Group by filesystem type
-    fs_types = defaultdict(
-        lambda: {
-            "configs": [],
-            "baseline_insert": [],
-            "dev_insert": [],
-            "baseline_query": [],
-            "dev_query": [],
-        }
-    )
-
-    for result in results:
-        fs = result.get("filesystem", "unknown")
-        block_size = result.get("block_size", "default")
-        config = f"{block_size}"
-
-        if config not in fs_types[fs]["configs"]:
-            fs_types[fs]["configs"].append(config)
-            fs_types[fs]["baseline_insert"].append(0)
-            fs_types[fs]["dev_insert"].append(0)
-            fs_types[fs]["baseline_query"].append(0)
-            fs_types[fs]["dev_query"].append(0)
-
-        idx = fs_types[fs]["configs"].index(config)
-
-        # Calculate average query QPS from all test configurations
-        query_qps = 0
-        if "query_performance" in result:
-            qp = result["query_performance"]
-            total_qps = 0
-            count = 0
-            for topk_key in ["topk_1", "topk_10", "topk_100"]:
-                if topk_key in qp:
-                    for batch_key in ["batch_1", "batch_10", "batch_100"]:
-                        if batch_key in qp[topk_key]:
-                            total_qps += qp[topk_key][batch_key].get(
-                                "queries_per_second", 0
-                            )
-                            count += 1
-            if count > 0:
-                query_qps = total_qps / count
-
-        if result.get("is_dev", False):
-            if "insert_performance" in result:
-                fs_types[fs]["dev_insert"][idx] = result["insert_performance"].get(
-                    "vectors_per_second", 0
-                )
-            fs_types[fs]["dev_query"][idx] = query_qps
-        else:
-            if "insert_performance" in result:
-                fs_types[fs]["baseline_insert"][idx] = result["insert_performance"].get(
-                    "vectors_per_second", 0
-                )
-            fs_types[fs]["baseline_query"][idx] = query_qps
-
-    # Create separate plots for each filesystem
-    for fs, data in fs_types.items():
-        if not data["configs"]:
-            continue
-
-        fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 10))
-
-        x = range(len(data["configs"]))
-
-        # Insert performance
-        ax1.plot(
-            x,
-            data["baseline_insert"],
-            "o-",
-            label="Baseline",
-            linewidth=2,
-            markersize=8,
-        )
-        ax1.plot(
-            x, data["dev_insert"], "s-", label="Development", linewidth=2, markersize=8
-        )
-        ax1.set_xlabel("Configuration")
-        ax1.set_ylabel("Insert QPS")
-        ax1.set_title(f"{fs.upper()} Insert Performance")
-        ax1.set_xticks(x)
-        ax1.set_xticklabels(data["configs"])
-        ax1.legend()
+    # Check if we have multi-filesystem data
+    if len(fs_performance) > 1:
+        # Multi-filesystem mode: separate lines for each filesystem
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        
+        colors = ["b", "r", "g", "m", "c", "y", "k"]
+        color_idx = 0
+        
+        for config_key, perf_data in fs_performance.items():
+            if not perf_data["insert_rates"]:
+                continue
+                
+            color = colors[color_idx % len(colors)]
+            iterations = list(range(1, len(perf_data["insert_rates"]) + 1))
+            
+            # Plot insert rate  
+            ax1.plot(
+                iterations,
+                perf_data["insert_rates"], 
+                f"{color}-o",
+                linewidth=2,
+                markersize=6,
+                label=config_key.upper(),
+            )
+            
+            # Plot insert time
+            ax2.plot(
+                iterations,
+                perf_data["insert_times"],
+                f"{color}-o", 
+                linewidth=2,
+                markersize=6,
+                label=config_key.upper(),
+            )
+            
+            color_idx += 1
+            
+        ax1.set_xlabel("Iteration")
+        ax1.set_ylabel("Vectors/Second")
+        ax1.set_title("Milvus Insert Rate by Storage Filesystem")
         ax1.grid(True, alpha=0.3)
-
-        # Query performance
-        ax2.plot(
-            x, data["baseline_query"], "o-", label="Baseline", linewidth=2, markersize=8
-        )
-        ax2.plot(
-            x, data["dev_query"], "s-", label="Development", linewidth=2, markersize=8
-        )
-        ax2.set_xlabel("Configuration")
-        ax2.set_ylabel("Query QPS")
-        ax2.set_title(f"{fs.upper()} Query Performance")
-        ax2.set_xticks(x)
-        ax2.set_xticklabels(data["configs"])
-        ax2.legend()
+        ax1.legend()
+        
+        ax2.set_xlabel("Iteration")
+        ax2.set_ylabel("Total Time (seconds)")
+        ax2.set_title("Milvus Insert Time by Storage Filesystem")
         ax2.grid(True, alpha=0.3)
-
-        plt.tight_layout()
-        plt.savefig(os.path.join(output_dir, f"{fs}_performance_trends.png"), dpi=150)
-        plt.close()
+        ax2.legend()
+    else:
+        # Single filesystem mode: original behavior
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+        
+        # Extract insert data from single filesystem
+        config_key = list(fs_performance.keys())[0] if fs_performance else None
+        if config_key:
+            perf_data = fs_performance[config_key]
+            iterations = list(range(1, len(perf_data["insert_rates"]) + 1))
+            
+            # Plot insert rate
+            ax1.plot(
+                iterations,
+                perf_data["insert_rates"],
+                "b-o",
+                linewidth=2,
+                markersize=6,
+            )
+            ax1.set_xlabel("Iteration")
+            ax1.set_ylabel("Vectors/Second") 
+            ax1.set_title("Vector Insert Rate Performance")
+            ax1.grid(True, alpha=0.3)
+            
+            # Plot insert time
+            ax2.plot(
+                iterations,
+                perf_data["insert_times"],
+                "r-o",
+                linewidth=2,
+                markersize=6,
+            )
+            ax2.set_xlabel("Iteration")
+            ax2.set_ylabel("Total Time (seconds)")
+            ax2.set_title("Vector Insert Time Performance") 
+            ax2.grid(True, alpha=0.3)
+            
+    plt.tight_layout()
+    plt.savefig(os.path.join(output_dir, "performance_trends.png"), dpi=150)
+    plt.close()
 
 
-def create_simple_performance_trends(results, output_dir):
-    """Create a simple performance trends chart for basic Milvus testing"""
+def create_heatmap_analysis(results, output_dir):
+    """Create multi-filesystem heatmap showing query performance"""
     if not results:
         return
-    
-    # Separate baseline and dev results
-    baseline_results = [r for r in results if not r.get("is_dev", False)]
-    dev_results = [r for r in results if r.get("is_dev", False)]
-    
-    if not baseline_results and not dev_results:
-        return
-    
-    fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 10))
-    
-    # Prepare data
-    baseline_insert = []
-    baseline_query = []
-    dev_insert = []
-    dev_query = []
-    labels = []
-    
-    # Process baseline results
-    for i, result in enumerate(baseline_results):
-        if "insert_performance" in result:
-            baseline_insert.append(result["insert_performance"].get("vectors_per_second", 0))
-        else:
-            baseline_insert.append(0)
+
+    # Group data by filesystem configuration
+    fs_performance = defaultdict(lambda: {
+        "query_data": [],
+        "config_key": "",
+    })
+
+    for result in results:
+        fs_type, block_size, config_key = _extract_filesystem_config(result)
 
-        # Calculate average query QPS
-        query_qps = 0
-        if "query_performance" in result:
-            qp = result["query_performance"]
-            total_qps = 0
-            count = 0
-            for topk_key in ["topk_1", "topk_10", "topk_100"]:
-                if topk_key in qp:
-                    for batch_key in ["batch_1", "batch_10", "batch_100"]:
-                        if batch_key in qp[topk_key]:
-                            total_qps += qp[topk_key][batch_key].get("queries_per_second", 0)
-                            count += 1
-            if count > 0:
-                query_qps = total_qps / count
-        baseline_query.append(query_qps)
-        labels.append(f"Run {i+1}")
-    
-    # Process dev results
-    for result in dev_results:
-        if "insert_performance" in result:
-            dev_insert.append(result["insert_performance"].get("vectors_per_second", 0))
-        else:
-            dev_insert.append(0)
+        query_perf = result.get("query_performance", {})
+        for topk, topk_data in query_perf.items():
+            for batch, batch_data in topk_data.items():
+                qps = batch_data.get("queries_per_second", 0)
+                fs_performance[config_key]["query_data"].append({
+                    "topk": topk,
+                    "batch": batch,
+                    "qps": qps,
+                })
+                fs_performance[config_key]["config_key"] = config_key
+
+    # Check if we have multi-filesystem data
+    if len(fs_performance) > 1:
+        # Multi-filesystem mode: separate heatmaps for each filesystem
+        num_fs = len(fs_performance)
+        fig, axes = plt.subplots(1, num_fs, figsize=(5*num_fs, 6))
+        if num_fs == 1:
+            axes = [axes]
 
-        query_qps = 0
-        if "query_performance" in result:
-            qp = result["query_performance"]
-            total_qps = 0
-            count = 0
-            for topk_key in ["topk_1", "topk_10", "topk_100"]:
-                if topk_key in qp:
-                    for batch_key in ["batch_1", "batch_10", "batch_100"]:
-                        if batch_key in qp[topk_key]:
-                            total_qps += qp[topk_key][batch_key].get("queries_per_second", 0)
-                            count += 1
-            if count > 0:
-                query_qps = total_qps / count
-        dev_query.append(query_qps)
-    
-    x = range(len(baseline_results) if baseline_results else len(dev_results))
-    
-    # Insert performance
-    if baseline_insert:
-        ax1.plot(x, baseline_insert, "o-", label="Baseline", linewidth=2, markersize=8)
-    if dev_insert:
-        ax1.plot(x[:len(dev_insert)], dev_insert, "s-", label="Development", linewidth=2, markersize=8)
-    ax1.set_xlabel("Test Run")
-    ax1.set_ylabel("Insert QPS")
-    ax1.set_title("Milvus Insert Performance")
-    ax1.set_xticks(x)
-    ax1.set_xticklabels(labels if labels else [f"Run {i+1}" for i in x])
-    ax1.legend()
-    ax1.grid(True, alpha=0.3)
-    
-    # Query performance
-    if baseline_query:
-        ax2.plot(x, baseline_query, "o-", label="Baseline", linewidth=2, markersize=8)
-    if dev_query:
-        ax2.plot(x[:len(dev_query)], dev_query, "s-", label="Development", linewidth=2, markersize=8)
-    ax2.set_xlabel("Test Run")
-    ax2.set_ylabel("Query QPS")
-    ax2.set_title("Milvus Query Performance")
-    ax2.set_xticks(x)
-    ax2.set_xticklabels(labels if labels else [f"Run {i+1}" for i in x])
-    ax2.legend()
-    ax2.grid(True, alpha=0.3)
+        # Define common structure for consistency
+        topk_order = ["topk_1", "topk_10", "topk_100"]
+        batch_order = ["batch_1", "batch_10", "batch_100"]
+        
+        for idx, (config_key, perf_data) in enumerate(fs_performance.items()):
+            # Create matrix for this filesystem
+            matrix = np.zeros((len(topk_order), len(batch_order)))
+            
+            # Fill matrix with data
+            query_dict = {}
+            for item in perf_data["query_data"]:
+                query_dict[(item["topk"], item["batch"])] = item["qps"]
+                
+            for i, topk in enumerate(topk_order):
+                for j, batch in enumerate(batch_order):
+                    matrix[i, j] = query_dict.get((topk, batch), 0)
+            
+            # Plot heatmap
+            im = axes[idx].imshow(matrix, cmap='viridis', aspect='auto')
+            axes[idx].set_title(f"{config_key.upper()} Query Performance")
+            axes[idx].set_xticks(range(len(batch_order)))
+            axes[idx].set_xticklabels([b.replace("batch_", "Batch ") for b in batch_order])
+            axes[idx].set_yticks(range(len(topk_order)))
+            axes[idx].set_yticklabels([t.replace("topk_", "Top-") for t in topk_order])
+            
+            # Add text annotations
+            for i in range(len(topk_order)):
+                for j in range(len(batch_order)):
+                    axes[idx].text(j, i, f'{matrix[i, j]:.0f}',
+                                 ha="center", va="center", color="white", fontweight="bold")
+            
+            # Add colorbar
+            cbar = plt.colorbar(im, ax=axes[idx])
+            cbar.set_label('Queries Per Second (QPS)')
+    else:
+        # Single filesystem mode
+        fig, ax = plt.subplots(1, 1, figsize=(8, 6))
+        
+        if fs_performance:
+            config_key = list(fs_performance.keys())[0]
+            perf_data = fs_performance[config_key]
+            
+            # Create matrix
+            topk_order = ["topk_1", "topk_10", "topk_100"]
+            batch_order = ["batch_1", "batch_10", "batch_100"]
+            matrix = np.zeros((len(topk_order), len(batch_order)))
+            
+            # Fill matrix with data
+            query_dict = {}
+            for item in perf_data["query_data"]:
+                query_dict[(item["topk"], item["batch"])] = item["qps"]
+                
+            for i, topk in enumerate(topk_order):
+                for j, batch in enumerate(batch_order):
+                    matrix[i, j] = query_dict.get((topk, batch), 0)
+            
+            # Plot heatmap
+            im = ax.imshow(matrix, cmap='viridis', aspect='auto')
+            ax.set_title("Milvus Query Performance Heatmap")
+            ax.set_xticks(range(len(batch_order)))
+            ax.set_xticklabels([b.replace("batch_", "Batch ") for b in batch_order])
+            ax.set_yticks(range(len(topk_order)))
+            ax.set_yticklabels([t.replace("topk_", "Top-") for t in topk_order])
+            
+            # Add text annotations
+            for i in range(len(topk_order)):
+                for j in range(len(batch_order)):
+                    ax.text(j, i, f'{matrix[i, j]:.0f}',
+                           ha="center", va="center", color="white", fontweight="bold")
+            
+            # Add colorbar
+            cbar = plt.colorbar(im, ax=ax)
+            cbar.set_label('Queries Per Second (QPS)')
 
     plt.tight_layout()
-    plt.savefig(os.path.join(output_dir, "performance_trends.png"), dpi=150)
+    plt.savefig(os.path.join(output_dir, "performance_heatmap.png"), dpi=150, bbox_inches="tight")
     plt.close()
 
 
-def generate_summary_statistics(results, output_dir):
-    """Generate summary statistics and save to JSON"""
-    summary = {
-        "total_tests": len(results),
-        "filesystems_tested": list(
-            set(r.get("filesystem", "unknown") for r in results)
-        ),
-        "configurations": {},
-        "performance_summary": {
-            "best_insert_qps": {"value": 0, "config": ""},
-            "best_query_qps": {"value": 0, "config": ""},
-            "average_insert_qps": 0,
-            "average_query_qps": 0,
-        },
-    }
-
-    # Calculate statistics
-    all_insert_qps = []
-    all_query_qps = []
-
-    for result in results:
-        fs = result.get("filesystem", "unknown")
-        block_size = result.get("block_size", "default")
-        is_dev = "dev" if result.get("is_dev", False) else "baseline"
-        config_name = f"{fs}-{block_size}-{is_dev}"
-
-        # Get actual performance metrics
-        if "insert_performance" in result:
-            insert_qps = result["insert_performance"].get("vectors_per_second", 0)
-        else:
-            insert_qps = 0
-
-        # Calculate average query QPS
-        query_qps = 0
-        if "query_performance" in result:
-            qp = result["query_performance"]
-            total_qps = 0
-            count = 0
-            for topk_key in ["topk_1", "topk_10", "topk_100"]:
-                if topk_key in qp:
-                    for batch_key in ["batch_1", "batch_10", "batch_100"]:
-                        if batch_key in qp[topk_key]:
-                            total_qps += qp[topk_key][batch_key].get(
-                                "queries_per_second", 0
-                            )
-                            count += 1
-            if count > 0:
-                query_qps = total_qps / count
-
-        all_insert_qps.append(insert_qps)
-        all_query_qps.append(query_qps)
-
-        summary["configurations"][config_name] = {
-            "insert_qps": insert_qps,
-            "query_qps": query_qps,
-            "host": result.get("host", "unknown"),
-        }
-
-        if insert_qps > summary["performance_summary"]["best_insert_qps"]["value"]:
-            summary["performance_summary"]["best_insert_qps"] = {
-                "value": insert_qps,
-                "config": config_name,
-            }
-
-        if query_qps > summary["performance_summary"]["best_query_qps"]["value"]:
-            summary["performance_summary"]["best_query_qps"] = {
-                "value": query_qps,
-                "config": config_name,
-            }
-
-    summary["performance_summary"]["average_insert_qps"] = (
-        np.mean(all_insert_qps) if all_insert_qps else 0
-    )
-    summary["performance_summary"]["average_query_qps"] = (
-        np.mean(all_query_qps) if all_query_qps else 0
-    )
-
-    # Save summary
-    with open(os.path.join(output_dir, "summary.json"), "w") as f:
-        json.dump(summary, f, indent=2)
-
-    return summary
-
-
 def main():
     if len(sys.argv) < 3:
         print("Usage: generate_graphs.py <results_dir> <output_dir>")
@@ -642,37 +340,23 @@ def main():
     results_dir = sys.argv[1]
     output_dir = sys.argv[2]
 
-    # Create output directory
+    # Ensure output directory exists
     os.makedirs(output_dir, exist_ok=True)
 
     # Load results
     results = load_results(results_dir)
-
     if not results:
-        print("No results found to analyze")
+        print(f"No valid results found in {results_dir}")
         sys.exit(1)
 
     print(f"Loaded {len(results)} result files")
 
     # Generate graphs
-    print("Generating performance heatmap...")
-    create_heatmap_analysis(results, output_dir)
-
-    print("Generating performance trends...")
     create_simple_performance_trends(results, output_dir)
+    create_heatmap_analysis(results, output_dir)
 
-    print("Generating summary statistics...")
-    summary = generate_summary_statistics(results, output_dir)
-
-    print(f"\nAnalysis complete! Graphs saved to {output_dir}")
-    print(f"Total configurations tested: {summary['total_tests']}")
-    print(
-        f"Best insert QPS: {summary['performance_summary']['best_insert_qps']['value']} ({summary['performance_summary']['best_insert_qps']['config']})"
-    )
-    print(
-        f"Best query QPS: {summary['performance_summary']['best_query_qps']['value']} ({summary['performance_summary']['best_query_qps']['config']})"
-    )
+    print(f"Graphs generated in {output_dir}")
 
 
 if __name__ == "__main__":
-    main()
+    main()
\ No newline at end of file
playbooks/roles/ai_collect_results/files/generate_html_report.py +223 -40
index a205577c..01ec734c 100755
@@ -69,6 +69,24 @@ HTML_TEMPLATE = """
             color: #7f8c8d;
             font-size: 0.9em;
         }}
+        .config-box {{
+            background: #f8f9fa;
+            border-left: 4px solid #3498db;
+            padding: 15px;
+            margin: 20px 0;
+            border-radius: 4px;
+        }}
+        .config-box h3 {{
+            margin-top: 0;
+            color: #2c3e50;
+        }}
+        .config-box ul {{
+            margin: 10px 0;
+            padding-left: 20px;
+        }}
+        .config-box li {{
+            margin: 5px 0;
+        }}
         .section {{
             background: white;
             padding: 30px;
@@ -162,15 +180,16 @@ HTML_TEMPLATE = """
 </head>
 <body>
     <div class="header">
-        <h1>AI Vector Database Benchmark Results</h1>
+        <h1>Milvus Vector Database Benchmark Results</h1>
         <div class="subtitle">Generated on {timestamp}</div>
     </div>
 
     <nav class="navigation">
         <ul>
             <li><a href="#summary">Summary</a></li>
+            {filesystem_nav_items}
             <li><a href="#performance-metrics">Performance Metrics</a></li>
-            <li><a href="#performance-trends">Performance Trends</a></li>
+            <li><a href="#performance-heatmap">Performance Heatmap</a></li>
             <li><a href="#detailed-results">Detailed Results</a></li>
         </ul>
     </nav>
@@ -192,34 +211,40 @@ HTML_TEMPLATE = """
             <div class="label">{best_query_config}</div>
         </div>
         <div class="card">
-            <h3>Test Runs</h3>
-            <div class="value">{total_tests}</div>
-            <div class="label">Benchmark Executions</div>
+            <h3>{fourth_card_title}</h3>
+            <div class="value">{fourth_card_value}</div>
+            <div class="label">{fourth_card_label}</div>
         </div>
     </div>
 
-    <div id="performance-metrics" class="section">
-        <h2>Performance Metrics</h2>
-        <p>Key performance indicators for Milvus vector database operations.</p>
+    {filesystem_comparison_section}
+    
+    {block_size_analysis_section}
+    
+    <div id="performance-heatmap" class="section">
+        <h2>Performance Heatmap</h2>
+        <p>Heatmap visualization showing performance metrics across all tested configurations.</p>
         <div class="graph-container">
-            <img src="graphs/performance_heatmap.png" alt="Performance Metrics">
+            <img src="graphs/performance_heatmap.png" alt="Performance Heatmap">
         </div>
     </div>
 
-    <div id="performance-trends" class="section">
-        <h2>Performance Trends</h2>
-        <p>Performance comparison between baseline and development configurations.</p>
-        <div class="graph-container">
-            <img src="graphs/performance_trends.png" alt="Performance Trends">
+    <div id="performance-metrics" class="section">
+        <h2>Performance Metrics</h2>
+        {config_summary}
+        <div class="graph-grid">
+            {performance_trend_graphs}
         </div>
     </div>
 
     <div id="detailed-results" class="section">
-        <h2>Detailed Results Table</h2>
+        <h2>Milvus Performance by Storage Filesystem</h2>
+        <p>This table shows how Milvus vector database performs when its data is stored on different filesystem types and configurations.</p>
         <table class="results-table">
             <thead>
                 <tr>
-                    <th>Host</th>
+                    <th>Filesystem</th>
+                    <th>Configuration</th>
                     <th>Type</th>
                     <th>Insert QPS</th>
                     <th>Query QPS</th>
@@ -260,51 +285,77 @@ def load_results(results_dir):
                 data = json.load(f)
                 # Get filesystem from JSON data first, then fallback to filename parsing
                 filename = os.path.basename(json_file)
-                
+
                 # Skip results without valid performance data
                 insert_perf = data.get("insert_performance", {})
                 query_perf = data.get("query_performance", {})
                 if not insert_perf or not query_perf:
                     continue
-                
+
                 # Get filesystem from JSON data
                 fs_type = data.get("filesystem", None)
-                
-                # If not in JSON, try to parse from filename (backwards compatibility)
-                if not fs_type and "debian13-ai" in filename:
-                    host_parts = (
-                        filename.replace("results_debian13-ai-", "")
-                        .replace("_1.json", "")
+
+                # Always try to parse from filename first since JSON data might be wrong
+                if "-ai-" in filename:
+                    # Handle both debian13-ai- and prod-ai- prefixes
+                    cleaned_filename = filename.replace("results_", "")
+
+                    # Extract the part after -ai-
+                    if "debian13-ai-" in cleaned_filename:
+                        host_part = cleaned_filename.replace("debian13-ai-", "")
+                    elif "prod-ai-" in cleaned_filename:
+                        host_part = cleaned_filename.replace("prod-ai-", "")
+                    else:
+                        # Generic extraction
+                        ai_index = cleaned_filename.find("-ai-")
+                        if ai_index != -1:
+                            host_part = cleaned_filename[ai_index + 4 :]  # Skip "-ai-"
+                        else:
+                            host_part = cleaned_filename
+
+                    # Remove file extensions and dev suffix
+                    host_part = (
+                        host_part.replace("_1.json", "")
                         .replace("_2.json", "")
                         .replace("_3.json", "")
-                        .split("-")
+                        .replace("-dev", "")
                     )
-                    if "xfs" in host_parts[0]:
+
+                    # Parse filesystem type and block size
+                    if host_part.startswith("xfs-"):
                         fs_type = "xfs"
-                        block_size = host_parts[1] if len(host_parts) > 1 else "4k"
-                    elif "ext4" in host_parts[0]:
+                        # Extract block size: xfs-4k-4ks -> 4k
+                        parts = host_part.split("-")
+                        if len(parts) >= 2:
+                            block_size = parts[1]  # 4k, 16k, 32k, 64k
+                        else:
+                            block_size = "4k"
+                    elif host_part.startswith("ext4-"):
                         fs_type = "ext4"
-                        block_size = host_parts[1] if len(host_parts) > 1 else "4k"
-                    elif "btrfs" in host_parts[0]:
+                        parts = host_part.split("-")
+                        block_size = parts[1] if len(parts) > 1 else "4k"
+                    elif host_part.startswith("btrfs"):
                         fs_type = "btrfs"
                         block_size = "default"
                     else:
-                        fs_type = "unknown"
-                        block_size = "unknown"
+                        # Fallback to JSON data if available
+                        if not fs_type:
+                            fs_type = "unknown"
+                            block_size = "unknown"
                 else:
                     # Set appropriate block size based on filesystem
                     if fs_type == "btrfs":
                         block_size = "default"
                     else:
                         block_size = data.get("block_size", "default")
-                
+
                 # Default to unknown if still not found
                 if not fs_type:
                     fs_type = "unknown"
                     block_size = "unknown"
-                
+
                 is_dev = "dev" in filename
-                
+
                 # Calculate average QPS from query performance data
                 query_qps = 0
                 query_count = 0
@@ -316,7 +367,7 @@ def load_results(results_dir):
                             query_count += 1
                 if query_count > 0:
                     query_qps = query_qps / query_count
-                
+
                 results.append(
                     {
                         "host": filename.replace("results_", "").replace(".json", ""),
@@ -348,12 +399,36 @@ def generate_table_rows(results, best_configs):
         if config_key in best_configs:
             row_class += " best-config"
 
+        # Generate descriptive labels showing Milvus is running on this filesystem
+        if result["filesystem"] == "xfs" and result["block_size"] != "default":
+            storage_label = f"XFS {result['block_size'].upper()}"
+            config_details = f"Block size: {result['block_size']}, Milvus data on XFS"
+        elif result["filesystem"] == "ext4":
+            storage_label = "EXT4"
+            if "bigalloc" in result.get("host", "").lower():
+                config_details = "EXT4 with bigalloc, Milvus data on ext4"
+            else:
+                config_details = (
+                    f"Block size: {result['block_size']}, Milvus data on ext4"
+                )
+        elif result["filesystem"] == "btrfs":
+            storage_label = "BTRFS"
+            config_details = "Default Btrfs settings, Milvus data on Btrfs"
+        else:
+            storage_label = result["filesystem"].upper()
+            config_details = f"Milvus data on {result['filesystem']}"
+
+        # Extract clean node identifier from hostname
+        node_name = result["host"].replace("results_", "").replace(".json", "")
+
         row = f"""
         <tr class="{row_class}">
-            <td>{result['host']}</td>
+            <td><strong>{storage_label}</strong></td>
+            <td>{config_details}</td>
             <td>{result['type']}</td>
             <td>{result['insert_qps']:,}</td>
             <td>{result['query_qps']:,}</td>
+            <td><code>{node_name}</code></td>
             <td>{result['timestamp']}</td>
         </tr>
         """
@@ -362,10 +437,66 @@ def generate_table_rows(results, best_configs):
     return "\n".join(rows)
 
 
+def generate_config_summary(results_dir):
+    """Generate configuration summary HTML from results"""
+    # Try to load first result file to get configuration
+    result_files = glob.glob(os.path.join(results_dir, "results_*.json"))
+    if not result_files:
+        return ""
+
+    try:
+        with open(result_files[0], "r") as f:
+            data = json.load(f)
+            config = data.get("config", {})
+
+            # Format configuration details
+            config_html = """
+        <div class="config-box">
+            <h3>Test Configuration</h3>
+            <ul>
+                <li><strong>Vector Dataset Size:</strong> {:,} vectors</li>
+                <li><strong>Vector Dimensions:</strong> {}</li>
+                <li><strong>Index Type:</strong> {} (M={}, ef_construction={}, ef={})</li>
+                <li><strong>Benchmark Runtime:</strong> {} seconds</li>
+                <li><strong>Batch Size:</strong> {:,}</li>
+                <li><strong>Test Iterations:</strong> {} runs with identical configuration</li>
+            </ul>
+        </div>
+            """.format(
+                config.get("vector_dataset_size", "N/A"),
+                config.get("vector_dimensions", "N/A"),
+                config.get("index_type", "N/A"),
+                config.get("index_hnsw_m", "N/A"),
+                config.get("index_hnsw_ef_construction", "N/A"),
+                config.get("index_hnsw_ef", "N/A"),
+                config.get("benchmark_runtime", "N/A"),
+                config.get("batch_size", "N/A"),
+                len(result_files),
+            )
+            return config_html
+    except Exception as e:
+        print(f"Warning: Could not generate config summary: {e}")
+        return ""
+
+
 def find_performance_trend_graphs(graphs_dir):
-    """Find performance trend graph"""
-    # Not used in basic implementation since we embed the graph directly
-    return ""
+    """Find performance trend graphs"""
+    graphs = []
+    # Look for filesystem-specific graphs in multi-fs mode
+    for fs in ["xfs", "ext4", "btrfs"]:
+        graph_path = f"{fs}_performance_trends.png"
+        if os.path.exists(os.path.join(graphs_dir, graph_path)):
+            graphs.append(
+                f'<div class="graph-container"><img src="graphs/{graph_path}" alt="{fs.upper()} Performance Trends"></div>'
+            )
+    # Fallback to simple performance trends for single mode
+    if not graphs and os.path.exists(
+        os.path.join(graphs_dir, "performance_trends.png")
+    ):
+        graphs.append(
+            '<div class="graph-container"><img src="graphs/performance_trends.png" alt="Performance Trends"></div>'
+        )
+    return "\n".join(graphs)
 
 
 def generate_html_report(results_dir, graphs_dir, output_path):
@@ -393,6 +524,50 @@ def generate_html_report(results_dir, graphs_dir, output_path):
     if summary["performance_summary"]["best_query_qps"]["config"]:
         best_configs.add(summary["performance_summary"]["best_query_qps"]["config"])
 
+    # Check if multi-filesystem testing is enabled (more than one filesystem)
+    filesystems_tested = summary.get("filesystems_tested", [])
+    is_multifs_enabled = len(filesystems_tested) > 1
+
+    # Generate conditional sections based on multi-fs status
+    if is_multifs_enabled:
+        filesystem_nav_items = """
+            <li><a href="#filesystem-comparison">Filesystem Comparison</a></li>
+            <li><a href="#block-size-analysis">Block Size Analysis</a></li>"""
+
+        filesystem_comparison_section = """<div id="filesystem-comparison" class="section">
+        <h2>Milvus Storage Filesystem Comparison</h2>
+        <p>Comparison of Milvus vector database performance when its data is stored on different filesystem types (XFS, ext4, Btrfs) with various configurations.</p>
+        <div class="graph-container">
+            <img src="graphs/filesystem_comparison.png" alt="Filesystem Comparison">
+        </div>
+    </div>"""
+
+        block_size_analysis_section = """<div id="block-size-analysis" class="section">
+        <h2>XFS Block Size Analysis</h2>
+        <p>Performance analysis of XFS filesystem with different block sizes (4K, 16K, 32K, 64K).</p>
+        <div class="graph-container">
+            <img src="graphs/xfs_block_size_analysis.png" alt="XFS Block Size Analysis">
+        </div>
+    </div>"""
+
+        # Multi-fs mode: show filesystem info
+        fourth_card_title = "Storage Filesystems"
+        fourth_card_value = str(len(filesystems_tested))
+        fourth_card_label = ", ".join(filesystems_tested).upper() + " for Milvus Data"
+    else:
+        # Single filesystem mode - hide multi-fs sections
+        filesystem_nav_items = ""
+        filesystem_comparison_section = ""
+        block_size_analysis_section = ""
+
+        # Single mode: show test iterations
+        fourth_card_title = "Test Iterations"
+        fourth_card_value = str(summary["total_tests"])
+        fourth_card_label = "Identical Configuration Runs"
+
+    # Generate configuration summary
+    config_summary = generate_config_summary(results_dir)
+
     # Generate HTML
     html_content = HTML_TEMPLATE.format(
         timestamp=datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
@@ -401,6 +576,14 @@ def generate_html_report(results_dir, graphs_dir, output_path):
         best_insert_config=summary["performance_summary"]["best_insert_qps"]["config"],
         best_query_qps=f"{summary['performance_summary']['best_query_qps']['value']:,}",
         best_query_config=summary["performance_summary"]["best_query_qps"]["config"],
+        fourth_card_title=fourth_card_title,
+        fourth_card_value=fourth_card_value,
+        fourth_card_label=fourth_card_label,
+        filesystem_nav_items=filesystem_nav_items,
+        filesystem_comparison_section=filesystem_comparison_section,
+        block_size_analysis_section=block_size_analysis_section,
+        config_summary=config_summary,
+        performance_trend_graphs=find_performance_trend_graphs(graphs_dir),
         table_rows=generate_table_rows(results, best_configs),
     )
 
playbooks/roles/ai_collect_results/tasks/main.yml +12 -30
index 6a15d89c..9586890a 100644
@@ -134,13 +134,22 @@
   ansible.builtin.command: >
     python3 {{ local_scripts_dir }}/analyze_results.py
     --results-dir {{ local_results_dir }}
-    --output-dir {{ local_results_dir }}
+    --output-dir {{ local_results_dir }}/graphs
     {% if ai_benchmark_enable_graphing | bool %}--config {{ local_scripts_dir }}/analysis_config.json{% endif %}
   register: analysis_result
   run_once: true
   delegate_to: localhost
   when: collected_results.files is defined and collected_results.files | length > 0
   tags: ['results', 'analysis']
+  failed_when: analysis_result.rc != 0
+
+- name: Display analysis script output
+  ansible.builtin.debug:
+    var: analysis_result
+  run_once: true
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_collect_results/templates/analysis_config.json.j2 +1 -1
index 5a879649..459cd602 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_milvus_storage/tasks/main.yml new file +161 -0
new file mode 100644
index 00000000..f8e4ea63

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml new file +279 -0
new file mode 100644
index 00000000..b4453b81

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_run/tasks/main.yml new file +23 -0
new file mode 100644
index 00000000..38dbba12

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml new file +104 -0
new file mode 100644
index 00000000..fd194550

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2 new file +42 -0
new file mode 100644
index 00000000..6216bf46

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_setup/defaults/main.yml new file +49 -0
new file mode 100644
index 00000000..c35d179f

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_multifs_setup/tasks/main.yml new file +70 -0
new file mode 100644
index 00000000..28f3ec40

Lines of this file are not shown. View the raw message.

playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py +107 -57
index 4ce14fb7..2aaa54ba 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/tasks/main.yml +19 -0
index 4b35d9f6..d36790b0 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/fstests.j2 +2 -0
index ac086c6e..32d90abf 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/gitr.j2 +2 -0
index 7f9094d4..3f30a5fb 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/hosts.j2 +13 -22
index cdcd1883..e9441605 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/nfstest.j2 +2 -0
index e427ac34..709d871d 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_hosts/templates/pynfs.j2 +2 -0
index 85c87dae..55add4d1 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/gen_nodes/tasks/main.yml +90 -0
index d54977be..b294d294 100644

Lines of this file are not shown. View the raw message.

playbooks/roles/guestfs/tasks/bringup/main.yml +15 -0
index c131de25..bd9f5260 100644

Lines of this file are not shown. View the raw message.

scripts/guestfs.Makefile +1 -1
index bd03f58c..f6c350a4 100644

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig +13 -0
index 2ffc6b65..d04570d8 100644

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig.fs new file +118 -0
new file mode 100644
index 00000000..a95d02c6

Lines of this file are not shown. View the raw message.

workflows/ai/Kconfig.multifs new file +184 -0
new file mode 100644
index 00000000..2b72dd6c

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/analysis_config.json +1 -1
index 2f90f4d5..5f0a9328 100644

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/analyze_results.py +927 -205
index 3d11fb11..2dc4a1d6 100755

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/generate_graphs.py +280 -1092
index 2e183e86..fafc62bf 100755

Lines of this file are not shown. View the raw message.

workflows/ai/scripts/generate_html_report.py +73 -21
index 3aa8342f..01ec734c 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
Chuck Lever

Part of the diff is not shown here. View the raw message.

#4 lore raw

Re: [PATCH 2/2] ai: add multi-filesystem testing support for Milvus benchmarks

Quoted 39 lines · Chuck Lever
Quoted 38 lines · Luis Chamberlain

Extend the AI workflow to support testing Milvus across multiple filesystem configurations simultaneously. This enables comprehensive performance comparisons between different filesystems and their configuration options.

Key features:
- Dynamic node generation based on enabled filesystem configurations
- Support for XFS, EXT4, and BTRFS with various mount options
- Per-filesystem result collection and analysis
- A/B testing across all filesystem configurations
- Automated comparison graphs between filesystems

Filesystem configurations:
- XFS: default, nocrc, bigtime with various block sizes (512, 1k, 2k, 4k)
- EXT4: default, nojournal, bigalloc configurations
- BTRFS: default, zlib, lzo, zstd compression options

Defconfigs:
- ai-milvus-multifs: Test 7 filesystem configs with A/B testing
- ai-milvus-multifs-distro: Test with distribution kernels
- ai-milvus-multifs-extended: Extended configs (14 filesystems total)

Node generation:
The system dynamically generates nodes based on enabled filesystem configurations. With A/B testing enabled, this creates baseline and dev nodes for each filesystem (e.g., debian13-ai-xfs-4k and debian13-ai-xfs-4k-dev).

Usage:
make defconfig-ai-milvus-multifs

  make bringup    # Creates nodes for each filesystem
  make ai         # Setup infrastructure on all nodes
  make ai-tests   # Run benchmarks on all filesystems

make ai-results # Collect and compare results

This enables systematic evaluation of how different filesystems and their configurations affect vector database performance.

Quoted 8 lines
Quoted 2 lines
Generated-by Claude AI

Hey Luis -

I'm looking at adding "AI optimized" and "GPU optimized" machine size choices in the cloud provider Kconfigs. I assume this set can take advantage of those. Any suggestions, or let me know if I'm way off base.

That would be next to add variability support, so patches welcomed on top! Some docs:

https://milvus.io/docs/v2.3.x/install_standalone-helm-gpu.md
https://milvus.io/docs/gpu_index.md
https://milvus.io/docs/gpu-cagra.md

Luis

#5 lore raw

Re: [PATCH 2/2] ai: add multi-filesystem testing support for Milvus benchmarks

In reply to Luis Chamberlain
Quoted 41 lines · Luis Chamberlain

Extend the AI workflow to support testing Milvus across multiple filesystem configurations simultaneously. This enables comprehensive performance comparisons between different filesystems and their configuration options.

Key features:
- Dynamic node generation based on enabled filesystem configurations
- Support for XFS, EXT4, and BTRFS with various mount options
- Per-filesystem result collection and analysis
- A/B testing across all filesystem configurations
- Automated comparison graphs between filesystems

Filesystem configurations:
- XFS: default, nocrc, bigtime with various block sizes (512, 1k, 2k, 4k)
- EXT4: default, nojournal, bigalloc configurations
- BTRFS: default, zlib, lzo, zstd compression options

Defconfigs:
- ai-milvus-multifs: Test 7 filesystem configs with A/B testing
- ai-milvus-multifs-distro: Test with distribution kernels
- ai-milvus-multifs-extended: Extended configs (14 filesystems total)

Node generation:
The system dynamically generates nodes based on enabled filesystem configurations. With A/B testing enabled, this creates baseline and dev nodes for each filesystem (e.g., debian13-ai-xfs-4k and debian13-ai-xfs-4k-dev).

Usage:
make defconfig-ai-milvus-multifs

  make bringup    # Creates nodes for each filesystem
  make ai         # Setup infrastructure on all nodes
  make ai-tests   # Run benchmarks on all filesystems

make ai-results # Collect and compare results

This enables systematic evaluation of how different filesystems and their configurations affect vector database performance.

Generated-by Claude AI

---

...

Quoted 351 lines
playbooks/roles/gen_hosts/tasks/main.yml +19 -0
index 4b35d9f6..d36790b0 100644
@@ -381,6 +381,25 @@
     - workflows_reboot_limit
     - ansible_hosts_template.stat.exists
 
+- name: Load AI nodes configuration for multi-filesystem setup
+  include_vars:
+    file: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    name: guestfs_nodes
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - ansible_hosts_template.stat.exists
+
+- name: Extract AI node names for multi-filesystem setup
+  set_fact:
+    all_generic_nodes: "{{ guestfs_nodes.guestfs_nodes | map(attribute='name') | list }}"
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - guestfs_nodes is defined
+
 - name: Generate the Ansible hosts file for a dedicated AI setup
   tags: ['hosts']
   ansible.builtin.template:
playbooks/roles/gen_hosts/templates/fstests.j2 +2 -0
index ac086c6e..32d90abf 100644
@@ -70,6 +70,7 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [krb5:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
+{% if kdevops_enable_iscsi or kdevops_nfsd_enable or kdevops_smbd_enable or kdevops_krb5_enable %}
 [service]
 {% if kdevops_enable_iscsi %}
 {{ kdevops_hosts_prefix }}-iscsi
@@ -85,3 +86,4 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
 [service:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+{% endif %}
playbooks/roles/gen_hosts/templates/gitr.j2 +2 -0
index 7f9094d4..3f30a5fb 100644
@@ -38,6 +38,7 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [nfsd:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
+{% if kdevops_enable_iscsi or kdevops_nfsd_enable %}
 [service]
 {% if kdevops_enable_iscsi %}
 {{ kdevops_hosts_prefix }}-iscsi
@@ -47,3 +48,4 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
 [service:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+{% endif %}
playbooks/roles/gen_hosts/templates/hosts.j2 +13 -22
index cdcd1883..e9441605 100644
@@ -119,39 +119,30 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
 [ai:vars]
 ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
 
-{% set fs_configs = [] %}
+{# Individual section groups for multi-filesystem testing #}
+{% set section_names = [] %}
 {% for node in all_generic_nodes %}
-{% set node_parts = node.split('-') %}
-{% if node_parts|length >= 3 %}
-{% set fs_type = node_parts[2] %}
-{% set fs_config = node_parts[3:] | select('ne', 'dev') | join('_') %}
-{% set fs_group = fs_type + '_' + fs_config if fs_config else fs_type %}
-{% if fs_group not in fs_configs %}
-{% set _ = fs_configs.append(fs_group) %}
+{% if not node.endswith('-dev') %}
+{% set section = node.replace(kdevops_host_prefix + '-ai-', '') %}
+{% if section != kdevops_host_prefix + '-ai' %}
+{% if section_names.append(section) %}{% endif %}
 {% endif %}
 {% endif %}
 {% endfor %}
 
-{% for fs_group in fs_configs %}
-[ai_{{ fs_group }}]
-{% for node in all_generic_nodes %}
-{% set node_parts = node.split('-') %}
-{% if node_parts|length >= 3 %}
-{% set fs_type = node_parts[2] %}
-{% set fs_config = node_parts[3:] | select('ne', 'dev') | join('_') %}
-{% set node_fs_group = fs_type + '_' + fs_config if fs_config else fs_type %}
-{% if node_fs_group == fs_group %}
-{{ node }}
-{% endif %}
+{% for section in section_names %}
+[ai_{{ section | replace('-', '_') }}]
+{{ kdevops_host_prefix }}-ai-{{ section }}
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-ai-{{ section }}-dev
 {% endif %}
-{% endfor %}
 
-[ai_{{ fs_group }}:vars]
+[ai_{{ section | replace('-', '_') }}:vars]
 ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
 
 {% endfor %}
 {% else %}
-{# Single-node AI hosts #}
+{# Single filesystem hosts (original behavior) #}
 [all]
 localhost ansible_connection=local
 {{ kdevops_host_prefix }}-ai
playbooks/roles/gen_hosts/templates/nfstest.j2 +2 -0
index e427ac34..709d871d 100644
@@ -38,6 +38,7 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [nfsd:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
+{% if kdevops_enable_iscsi or kdevops_nfsd_enable %}
 [service]
 {% if kdevops_enable_iscsi %}
 {{ kdevops_hosts_prefix }}-iscsi
@@ -47,3 +48,4 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% endif %}
 [service:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+{% endif %}
playbooks/roles/gen_hosts/templates/pynfs.j2 +2 -0
index 85c87dae..55add4d1 100644
@@ -23,6 +23,7 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {{ kdevops_hosts_prefix }}-nfsd
 [nfsd:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+{% if true %}
 [service]
 {% if kdevops_enable_iscsi %}
 {{ kdevops_hosts_prefix }}-iscsi
@@ -30,3 +31,4 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {{ kdevops_hosts_prefix }}-nfsd
 [service:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+{% endif %}
playbooks/roles/gen_nodes/tasks/main.yml +90 -0
index d54977be..b294d294 100644
@@ -658,6 +658,7 @@
     - kdevops_workflow_enable_ai
     - ansible_nodes_template.stat.exists
     - not kdevops_baseline_and_dev
+    - not ai_enable_multifs_testing|default(false)|bool
 
 - name: Generate the AI kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template
   tags: ['hosts']
@@ -675,6 +676,95 @@
     - kdevops_workflow_enable_ai
     - ansible_nodes_template.stat.exists
     - kdevops_baseline_and_dev
+    - not ai_enable_multifs_testing|default(false)|bool
+
+- name: Infer enabled AI multi-filesystem configurations
+  vars:
+    kdevops_config_data: "{{ lookup('file', topdir_path + '/.config') }}"
+    # Find all enabled AI multifs configurations
+    xfs_configs: >-
+      {{
+        kdevops_config_data | regex_findall('^CONFIG_AI_MULTIFS_XFS_(.*)=y$', multiline=True)
+        | map('lower')
+        | map('regex_replace', '_', '-')
+        | map('regex_replace', '^', 'xfs-')
+        | list
+        if kdevops_config_data | regex_search('^CONFIG_AI_MULTIFS_TEST_XFS=y$', multiline=True)
+        else []
+      }}
+    ext4_configs: >-
+      {{
+        kdevops_config_data | regex_findall('^CONFIG_AI_MULTIFS_EXT4_(.*)=y$', multiline=True)
+        | map('lower')
+        | map('regex_replace', '_', '-')
+        | map('regex_replace', '^', 'ext4-')
+        | list
+        if kdevops_config_data | regex_search('^CONFIG_AI_MULTIFS_TEST_EXT4=y$', multiline=True)
+        else []
+      }}
+    btrfs_configs: >-
+      {{
+        kdevops_config_data | regex_findall('^CONFIG_AI_MULTIFS_BTRFS_(.*)=y$', multiline=True)
+        | map('lower')
+        | map('regex_replace', '_', '-')
+        | map('regex_replace', '^', 'btrfs-')
+        | list
+        if kdevops_config_data | regex_search('^CONFIG_AI_MULTIFS_TEST_BTRFS=y$', multiline=True)
+        else []
+      }}
+  set_fact:
+    ai_multifs_enabled_configs: "{{ (xfs_configs + ext4_configs + btrfs_configs) | unique }}"
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - ansible_nodes_template.stat.exists
+
+- name: Create AI nodes for each filesystem configuration (no dev)
+  vars:
+    filesystem_nodes: "{{ [kdevops_host_prefix + '-ai-'] | product(ai_multifs_enabled_configs | default([])) | map('join') | list }}"
+  set_fact:
+    ai_enabled_section_types: "{{ filesystem_nodes }}"
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - ansible_nodes_template.stat.exists
+    - not kdevops_baseline_and_dev
+    - ai_multifs_enabled_configs is defined
+    - ai_multifs_enabled_configs | length > 0
+
+- name: Create AI nodes for each filesystem configuration with dev hosts
+  vars:
+    filesystem_nodes: "{{ [kdevops_host_prefix + '-ai-'] | product(ai_multifs_enabled_configs | default([])) | map('join') | list }}"
+  set_fact:
+    ai_enabled_section_types: "{{ filesystem_nodes | product(['', '-dev']) | map('join') | list }}"
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - ansible_nodes_template.stat.exists
+    - kdevops_baseline_and_dev
+    - ai_multifs_enabled_configs is defined
+    - ai_multifs_enabled_configs | length > 0
+
+- name: Generate the AI multi-filesystem kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
+  tags: [ 'hosts' ]
+  vars:
+    node_template: "{{ kdevops_nodes_template | basename }}"
+    nodes: "{{ ai_enabled_section_types | regex_replace('\\[') | regex_replace('\\]') | replace(\"'\", '') | split(', ') }}"
+    all_generic_nodes: "{{ ai_enabled_section_types }}"
+  template:
+    src: "{{ node_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    force: yes
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_ai
+    - ai_enable_multifs_testing|default(false)|bool
+    - ansible_nodes_template.stat.exists
+    - ai_enabled_section_types is defined
+    - ai_enabled_section_types | length > 0
 
 - name: Get the control host's timezone
   ansible.builtin.command: "timedatectl show -p Timezone --value"
playbooks/roles/guestfs/tasks/bringup/main.yml +15 -0
index c131de25..bd9f5260 100644
@@ -1,11 +1,16 @@
 ---
 - name: List defined libvirt guests
   run_once: true
+  delegate_to: localhost
   community.libvirt.virt:
     command: list_vms
     uri: "{{ libvirt_uri }}"
   register: defined_vms
 
+- name: Debug defined VMs
+  debug:
+    msg: "Hostname: {{ inventory_hostname }}, Defined VMs: {{ hostvars['localhost']['defined_vms']['list_vms'] | default([]) }}, Check: {{ inventory_hostname not in (hostvars['localhost']['defined_vms']['list_vms'] | default([])) }}"
+
 - name: Provision each target node
   when:
     - "inventory_hostname not in defined_vms.list_vms"
@@ -25,10 +30,13 @@
             path: "{{ ssh_key_dir }}"
             state: directory
             mode: "u=rwx"
+          delegate_to: localhost
 
         - name: Generate fresh keys for each target node
           ansible.builtin.command:
             cmd: 'ssh-keygen -q -t ed25519 -f {{ ssh_key }} -N ""'
+            creates: "{{ ssh_key }}"
+          delegate_to: localhost
 
     - name: Set the pathname of the root disk image for each target node
       ansible.builtin.set_fact:
@@ -38,15 +46,18 @@
       ansible.builtin.file:
         path: "{{ storagedir }}/{{ inventory_hostname }}"
         state: directory
+      delegate_to: localhost
 
     - name: Duplicate the root disk image for each target node
       ansible.builtin.command:
         cmd: "cp --reflink=auto {{ base_image }} {{ root_image }}"
+      delegate_to: localhost
 
     - name: Get the timezone of the control host
       ansible.builtin.command:
         cmd: "timedatectl show -p Timezone --value"
       register: host_timezone
+      delegate_to: localhost
 
     - name: Build the root image for each target node (as root)
       become: true
@@ -103,6 +114,7 @@
         name: "{{ inventory_hostname }}"
         xml: "{{ lookup('file', xml_file) }}"
         uri: "{{ libvirt_uri }}"
+      delegate_to: localhost
 
     - name: Find PCIe passthrough devices
       ansible.builtin.find:
@@ -110,6 +122,7 @@
         file_type: file
         patterns: "pcie_passthrough_*.xml"
       register: passthrough_devices
+      delegate_to: localhost
 
     - name: Attach PCIe passthrough devices to each target node
       environment:
@@ -124,6 +137,7 @@
       loop: "{{ passthrough_devices.files }}"
       loop_control:
         label: "Doing PCI-E passthrough for device {{ item }}"
+      delegate_to: localhost
       when:
         - passthrough_devices.matched > 0
 
@@ -142,3 +156,4 @@
     name: "{{ inventory_hostname }}"
     uri: "{{ libvirt_uri }}"
     state: running
+  delegate_to: localhost
scripts/guestfs.Makefile +0 -0
index bd03f58c..f6c350a4 100644

scripts/guestfs.Makefile bringup_guestfs: $(GUESTFS_BRINGUP_DEPS) · line 82

Quoted 5 lines
scripts/guestfs.Makefile +1 -1
@@ -79,7 +79,7 @@ bringup_guestfs: $(GUESTFS_BRINGUP_DEPS)
 		--extra-vars=@./extra_vars.yaml \
 		--tags network,pool,base_image
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
-		--limit 'baseline:dev:service' \
+		--limit 'baseline:dev:service:ai' \

I'm not sure if I understand the need of this new ai group. Can you clarify?

Why aren't baseline or dev groups sufficient for this AI workload? What's the role of this ai group?

Note: I kept the hunks above to make it easier to reference the part I believe is most relevant to my questions (hopefully).

#6 lore raw

Re: [PATCH 2/2] ai: add multi-filesystem testing support for Milvus benchmarks

Quoted 14 lines · Daniel Gomez

scripts/guestfs.Makefile bringup_guestfs: $(GUESTFS_BRINGUP_DEPS) · line 82

Quoted 11 lines
scripts/guestfs.Makefile +1 -1
index bd03f58c..f6c350a4 100644
@@ -79,7 +79,7 @@ bringup_guestfs: $(GUESTFS_BRINGUP_DEPS)
              --extra-vars=@./extra_vars.yaml \
              --tags network,pool,base_image
      $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
-             --limit 'baseline:dev:service' \
+             --limit 'baseline:dev:service:ai' \

I'm not sure if I understand the need of this new ai group. Can you clarify?

Quoted 5 lines

Why aren't baseline or dev groups sufficient for this AI workload? What's the role of this ai group?

Note: I kept the hunks above to make it easier to reference the part I believe is most relevant to my questions (hopefully).

No good reason that I can think of, we can nuke it.

Luis

#7 lore raw
In reply to Luis Chamberlain
Quoted 26 lines · Luis Chamberlain

This adds the ability to test milvus on minio with different filesystem configuration targets. There's a basic configuration you can run which will just support one default filesystem, which will be used where you place your docker image and also where we place the minio instance. Then there is multifs support where just as with fstests support on kdevops you can select a slew of different filesystem targets to try to test.

Recommendation is to stick to 40 iterations at 1,000,000 tests unless you have more than 100 GiB per guest to spare. If you have space to spare then you know how to ballpark it.

On High Capacity SSDs, the world is our oyster.

You can see a demo of results here:

https://github.com/mcgrof/demo-milvus-kdevops-results

These are just demos. On guests. Nothing really useful. I should point out this has AB testing automated as well so we can leverage this to test for instance ... parallel writeback in an automated way ;)

If you want to test this you can also use this branch on kdevops:

https://github.com/linux-kdevops/kdevops/tree/mcgrof/20250827-milvus

Quoted 9 lines

I am in hopes someone will just prompt an AI for bare metal support while I sleep. It should be... easy. Just create the partitions already, use one host and ask the prompt to not mkfs for you. So don't use multi-fs support at first. Just use the option to create the storage partition where you place docker. In fact you can copy and paste this prompt the the AI, and I think it will know what to do. You just skip some steps as the filesystems can be created and mounted for you. You just need the host file created by you manually for the target node. That and infer user and group id support (WORKFLOW_INFER_USER_AND_GROUP).

Pushed.

Luis